Ingestion
Ingestion on the Analytical Platform is currently a beta service.
Service Requirements
Required Information
To use the Ingestion service, data owners must provide the following information:
- Supplier’s name
- Supplier’s email
- Supplier’s IP address(es)
- Supplier’s SSH public key
- Target location on Analytical Platform (e.g.
s3://${TARGET_BUCKET}/${OPTIONAL_PREFIX}) - Target bucket’s KMS key if encrypted
Please raise a support ticket here with the required information to start the onboarding process.
Note: The Supplier’s name you provide us will become your
USERNAME.
Optional Information
- Slack channel
User with Egress
By default, Ingestion users can upload files only. You can also request to be added as a User with Egress, which allows you to both upload and download files via SFTP.
To request egress access, include the following additional information in your support ticket:
- Egress S3 bucket
- Egress KMS key
Note: Users with Egress are presented with two directories when connected to SFTP:
/uploadfor ingestion and/downloadfor egress.
Target Bucket Overview
The Target Bucket is the S3 bucket where transferred files will be delivered. Buckets can exist in any Ministry of Justice AWS account, but setup differs depending on ownership.
Target Bucket Scenarios
There are two main scenarios:
Buckets in analytical-platform-data-production, such as:
mojap-landmojap-land-dev
Buckets outside analytical-platform-data-production.
Who Is Responsible?
Ministry of Justice colleagues should raise the appropriate Pull Requests on the target bucket configuration, to enable the transfer. Third-party consumers who cannot raise Pull Requests can request help in the Slack channel: #ask-analytical-platform
Adding permissions to mojap-land or mojap-land-dev
The target S3 bucket (and if using SSE-KMS, the KMS key) must have the correct permissions to allow the final transfer Lambda function to copy files to it.
An example Pull Request showing how to add a bucket to mojap-land-dev.
Destination Bucket Permissions
For a given S3 bucket <destination-bucket-name> not located in analytical-platform-data-production, include the following statement:
{
"Version": "2012-10-17",
"Statement": [
...
{
"Sid": "AllowAnalyticalPlatformIngestionService",
"Effect": "Allow",
"Principal": {
"AWS": "arn:aws:iam::<ingestion-account-ID>:role/transfer"
},
"Action": [
"s3:DeleteObject",
"s3:GetObject",
"s3:GetObjectAcl",
"s3:PutObject",
"s3:PutObjectAcl",
"s3:PutObjectTagging"
],
"Resource": [
"arn:aws:s3:::<destination-bucket-name>",
"arn:aws:s3:::<destination-bucket-name>/*"
]
}
]
}
Use the correct ingestion-account-ID based on the environment:
- for development, use
730335344807 - for production, use
471112983409
Destination Bucket KMS key
If the destination S3 bucket is encrypted with a customer-managed KMS key, the Analytical Platform ingestion role must be allowed to encrypt objects with that key. Add the following statement to the KMS key policy for the bucket’s key.
{
"Sid": "AllowAnalyticalPlatformIngestionService",
"Effect": "Allow",
"Principal": {
"AWS": "arn:aws:iam::<ingestion-account-ID>:role/transfer"
},
"Action": [
"kms:GenerateDataKey",
"kms:Encrypt"
],
"Resource": "*"
}
Use the correct ingestion-account-ID based on the environment:
- for development, use
730335344807 - for production, use
471112983409
Connection Instructions
Connect to our ingestion service using the following commands:
Production
sftp -P 2222 ${USERNAME}@sftp.ingestion.analytical-platform.service.justice.gov.uk
Development
sftp -P 2222 ${USERNAME}@sftp.development.ingestion.analytical-platform.service.justice.gov.uk
User Home Directories and Access Permissions
Each user connecting to the ingestion service is assigned a dedicated home directory within the mojap-ingestion-<environment>-landing S3 bucket. The directory format is:
/mojap-ingestion-<environment>-landing/<username>
For example:
/mojap-ingestion-production-landing/analytical-platform
When users connect via their SFTP they are restricted to their assigned home directory. Attempts to list or read directories outside of this path (for example, the root /) will result in permission errors.
Known Limitations
File Names
File names with spaces included are not supported.
File Size Transfer Limit
The File Transfer Service supports files up to 5 GB in size.