Ingestion
Ingestion on the Analytical Platform is currently a beta service.
The ingestion service provides a secure and approved route for transferring data into the Ministry of Justice’s AWS environments.
It is primarily used to copy data into the Analytical Platform but can also transfer files to other authorised buckets in Ministry of Justice AWS accounts.
The ingestion service supports data transfers up to 5 GB in size. For larger files, please contact us at #ask-analytical-platform.
To use the ingestion service, you’ll need to give us the following:
- Supplier’s name (this can be a person or organisation)
- Supplier’s email
- Supplier’s IP address(es)
- Supplier’s SSH public key
- Destination bucket’s location on Analytical Platform (e.g.
s3://${DESTINATION_BUCKET}/${OPTIONAL_PREFIX}) - Destination bucket’s KMS key if encrypted
Please raise a support ticket here with the required information to start the onboarding process.
Note: We’ll use the supplier’s name as your
USERNAME.
Request access if you need to download files
By default, ingestion users can only upload files. If you need to download files too, raise a ticket to request Egress access which allows you to upload and download files using SFTP.
You’ll need to include your:
- Egress S3 bucket
- Egress KMS key
Note: Users with Egress are presented with two directories when connected to SFTP:
/uploadfor ingestion and/downloadfor Egress.
Choose your destination bucket
The ‘destination bucket’ is the S3 bucket where transferred files will be delivered. Buckets can exist in any Ministry of Justice AWS account, but setup differs depending on ownership.
There are two options for where to copy your data.
Option 1: Buckets in analytical-platform-data-production, such as:
mojap-landmojap-land-dev
Option 2: Buckets outside analytical-platform-data-production but within a Ministry of Justice AWS account.
Update destination bucket’s permissions
You’ll need to add the correct permissions to the destination bucket’s configuration by raising a pull request to update the bucket policy file.
If you are a member of the Ministry of Justice GitHub organisation, you should raise the pull requests yourself.
If you are a not member of the Ministry of Justice GitHub organisation, ask someone to raise pull requests on your behalf in the Slack channel #ask-analytical-platform.
Adding permissions to mojap-land or mojap-land-dev
The destination S3 bucket (and if using SSE-KMS, the KMS key) must have the correct permissions to allow the final transfer Lambda function to copy files to it.
An example Pull Request showing how to add a bucket to mojap-land-dev.
Destination bucket permissions
For a given S3 bucket <destination-bucket-name> not located in analytical-platform-data-production, include the following statement:
{
"Version": "2012-10-17",
"Statement": [
...
{
"Sid": "AllowAnalyticalPlatformIngestionService",
"Effect": "Allow",
"Principal": {
"AWS": "arn:aws:iam::<ingestion-account-ID>:role/transfer"
},
"Action": [
"s3:DeleteObject",
"s3:GetObject",
"s3:GetObjectAcl",
"s3:PutObject",
"s3:PutObjectAcl",
"s3:PutObjectTagging"
],
"Resource": [
"arn:aws:s3:::<destination-bucket-name>",
"arn:aws:s3:::<destination-bucket-name>/*"
]
}
]
}
Use the correct ingestion-account-ID based on the environment:
- for development, use
730335344807 - for production, use
471112983409
Destination bucket KMS key
If the destination S3 bucket is encrypted with a customer-managed KMS key, the Analytical Platform ingestion role must be allowed to encrypt objects with that key. Add the following statement to the KMS key policy for the bucket’s key.
{
"Sid": "AllowAnalyticalPlatformIngestionService",
"Effect": "Allow",
"Principal": {
"AWS": "arn:aws:iam::<ingestion-account-ID>:role/transfer"
},
"Action": [
"kms:GenerateDataKey",
"kms:Encrypt"
],
"Resource": "*"
}
Use the correct ingestion-account-ID based on the environment:
- for development, use
730335344807 - for production, use
471112983409
Connection Instructions
Connect to our ingestion service using the following commands (see note on USERNAME above):
Production
sftp -P 2222 ${USERNAME}@sftp.ingestion.analytical-platform.service.justice.gov.uk
Development
sftp -P 2222 ${USERNAME}@sftp.development.ingestion.analytical-platform.service.justice.gov.uk
User Home Directories and Access Permissions
Each user connecting to the ingestion service is assigned a dedicated home directory within the mojap-ingestion-<environment>-landing S3 bucket. The directory format is:
/mojap-ingestion-<environment>-landing/<username>
For example:
/mojap-ingestion-production-landing/analytical-platform
When users connect via their SFTP they are restricted to their assigned home directory. Attempts to list or read directories outside of this path (for example, the root /) will result in permission errors.
Known Limitations
File Names
You cannot use file names with spaces.