Copyright	(c) 2013-2023 Brendan Hay
License	Mozilla Public License, v. 2.0.
Maintainer	Brendan Hay
Stability	auto-generated
Portability	non-portable (GHC extensions)
Safe Haskell	Safe-Inferred
Language	Haskell2010

Amazonka.SageMaker.Types.S3DataSource

Description

Documentation

data S3DataSource Source #

Describes the S3 data source.

See: newS3DataSource smart constructor.

Constructors

S3DataSource'

Fields

attributeNames :: Maybe [Text]
A list of one or more attribute names to use that are found in a specified augmented manifest file.
instanceGroupNames :: Maybe [Text]
A list of names of instance groups that get data from the S3 data source.
s3DataDistributionType :: Maybe S3DataDistribution
If you want SageMaker to replicate the entire dataset on each ML compute instance that is launched for model training, specify FullyReplicated.
If you want SageMaker to replicate a subset of data on each ML compute instance that is launched for model training, specify ShardedByS3Key. If there are n ML compute instances launched for a training job, each instance gets approximately 1/n of the number of S3 objects. In this case, model training on each machine uses only the subset of training data.
Don't choose more ML compute instances for training than available S3 objects. If you do, some nodes won't get any data and you will pay for nodes that aren't getting any training data. This applies in both File and Pipe modes. Keep this in mind when developing algorithms.
In distributed training, where you use multiple ML compute EC2 instances, you might choose ShardedByS3Key. If the algorithm requires copying training data to the ML storage volume (when TrainingInputMode is set to File), this copies 1/n of the number of objects.
s3DataType :: S3DataType
If you choose S3Prefix, S3Uri identifies a key name prefix. SageMaker uses all objects that match the specified key name prefix for model training.
If you choose ManifestFile, S3Uri identifies an object that is a manifest file containing a list of object keys that you want SageMaker to use for model training.
If you choose AugmentedManifestFile, S3Uri identifies an object that is an augmented manifest file in JSON lines format. This file contains the data you want to use for model training. AugmentedManifestFile can only be used if the Channel's input mode is Pipe.
s3Uri :: Text
Depending on the value specified for the S3DataType, identifies either a key name prefix or a manifest. For example:
- A key name prefix might look like this: s3://bucketname/exampleprefix
- A manifest might look like this: s3://bucketname/example.manifest
  A manifest is an S3 object which is a JSON file consisting of an array of elements. The first element is a prefix which is followed by one or more suffixes. SageMaker appends the suffix elements to the prefix to get a full set of S3Uri. Note that the prefix must be a valid non-empty S3Uri that precludes users from specifying a manifest whose individual S3Uri is sourced from different S3 buckets.
  The following code example shows a valid manifest format:
```
[ {"prefix": "s3://customer_bucket/some/prefix/"},
```
```
 "relative/path/to/custdata-1",
```
```
 "relative/path/custdata-2",
```
```
 ...
```
```
 "relative/path/custdata-N"
```
```
]
```
  This JSON is equivalent to the following S3Uri list:
```
s3://customer_bucket/some/prefix/relative/path/to/custdata-1
```
```
s3://customer_bucket/some/prefix/relative/path/custdata-2
```
```
...
```
```
s3://customer_bucket/some/prefix/relative/path/custdata-N
```
  The complete set of S3Uri in this manifest is the input data for the channel for this data source. The object that each S3Uri points to must be readable by the IAM role that SageMaker uses to perform tasks on your behalf.

Instances

Instances details

FromJSON S3DataSource Source #
Instance details Defined in Amazonka.SageMaker.Types.S3DataSource Methods parseJSON :: Value -> Parser S3DataSource # parseJSONList :: Value -> Parser [S3DataSource] #
ToJSON S3DataSource Source #
Instance details Defined in Amazonka.SageMaker.Types.S3DataSource Methods toJSON :: S3DataSource -> Value # toEncoding :: S3DataSource -> Encoding # toJSONList :: [S3DataSource] -> Value # toEncodingList :: [S3DataSource] -> Encoding #
Generic S3DataSource Source #
Instance details Defined in Amazonka.SageMaker.Types.S3DataSource Associated Types type Rep S3DataSource :: Type -> Type # Methods from :: S3DataSource -> Rep S3DataSource x # to :: Rep S3DataSource x -> S3DataSource #
Read S3DataSource Source #
Instance details Defined in Amazonka.SageMaker.Types.S3DataSource Methods readsPrec :: Int -> ReadS S3DataSource # readList :: ReadS [S3DataSource] # readPrec :: ReadPrec S3DataSource # readListPrec :: ReadPrec [S3DataSource] #
Show S3DataSource Source #
Instance details Defined in Amazonka.SageMaker.Types.S3DataSource Methods showsPrec :: Int -> S3DataSource -> ShowS # show :: S3DataSource -> String # showList :: [S3DataSource] -> ShowS #
NFData S3DataSource Source #
Instance details Defined in Amazonka.SageMaker.Types.S3DataSource Methods rnf :: S3DataSource -> () #
Eq S3DataSource Source #
Instance details Defined in Amazonka.SageMaker.Types.S3DataSource Methods (==) :: S3DataSource -> S3DataSource -> Bool # (/=) :: S3DataSource -> S3DataSource -> Bool #
Hashable S3DataSource Source #
Instance details Defined in Amazonka.SageMaker.Types.S3DataSource Methods hashWithSalt :: Int -> S3DataSource -> Int # hash :: S3DataSource -> Int #
type Rep S3DataSource Source #
Instance details Defined in Amazonka.SageMaker.Types.S3DataSource type Rep S3DataSource = D1 ('MetaData "S3DataSource" "Amazonka.SageMaker.Types.S3DataSource" "amazonka-sagemaker-2.0-9SyrKZ4KqhsL1qX9u3ILA3" 'False) (C1 ('MetaCons "S3DataSource'" 'PrefixI 'True) ((S1 ('MetaSel ('Just "attributeNames") 'NoSourceUnpackedness 'NoSourceStrictness 'DecidedStrict) (Rec0 (Maybe [Text])) :: S1 ('MetaSel ('Just "instanceGroupNames") 'NoSourceUnpackedness 'NoSourceStrictness 'DecidedStrict) (Rec0 (Maybe [Text]))) :: (S1 ('MetaSel ('Just "s3DataDistributionType") 'NoSourceUnpackedness 'NoSourceStrictness 'DecidedStrict) (Rec0 (Maybe S3DataDistribution)) :: (S1 ('MetaSel ('Just "s3DataType") 'NoSourceUnpackedness 'NoSourceStrictness 'DecidedStrict) (Rec0 S3DataType) :: S1 ('MetaSel ('Just "s3Uri") 'NoSourceUnpackedness 'NoSourceStrictness 'DecidedStrict) (Rec0 Text)))))

newS3DataSource Source #

Arguments

:: S3DataType	`$sel:s3DataType:S3DataSource'`
-> Text	`$sel:s3Uri:S3DataSource'`
-> S3DataSource

Create a value of S3DataSource with all optional fields omitted.

Use generic-lens or optics to modify other optional fields.

The following record fields are available, with the corresponding lenses provided for backwards compatibility:

$sel:attributeNames:S3DataSource', s3DataSource_attributeNames - A list of one or more attribute names to use that are found in a specified augmented manifest file.

$sel:instanceGroupNames:S3DataSource', s3DataSource_instanceGroupNames - A list of names of instance groups that get data from the S3 data source.

$sel:s3DataDistributionType:S3DataSource', s3DataSource_s3DataDistributionType - If you want SageMaker to replicate the entire dataset on each ML compute instance that is launched for model training, specify FullyReplicated.

If you want SageMaker to replicate a subset of data on each ML compute instance that is launched for model training, specify ShardedByS3Key. If there are n ML compute instances launched for a training job, each instance gets approximately 1/n of the number of S3 objects. In this case, model training on each machine uses only the subset of training data.

Don't choose more ML compute instances for training than available S3 objects. If you do, some nodes won't get any data and you will pay for nodes that aren't getting any training data. This applies in both File and Pipe modes. Keep this in mind when developing algorithms.

In distributed training, where you use multiple ML compute EC2 instances, you might choose ShardedByS3Key. If the algorithm requires copying training data to the ML storage volume (when TrainingInputMode is set to File), this copies 1/n of the number of objects.

$sel:s3DataType:S3DataSource', s3DataSource_s3DataType - If you choose S3Prefix, S3Uri identifies a key name prefix. SageMaker uses all objects that match the specified key name prefix for model training.

If you choose ManifestFile, S3Uri identifies an object that is a manifest file containing a list of object keys that you want SageMaker to use for model training.

If you choose AugmentedManifestFile, S3Uri identifies an object that is an augmented manifest file in JSON lines format. This file contains the data you want to use for model training. AugmentedManifestFile can only be used if the Channel's input mode is Pipe.

$sel:s3Uri:S3DataSource', s3DataSource_s3Uri - Depending on the value specified for the S3DataType, identifies either a key name prefix or a manifest. For example:

A key name prefix might look like this: s3://bucketname/exampleprefix
A manifest might look like this: s3://bucketname/example.manifest
A manifest is an S3 object which is a JSON file consisting of an array of elements. The first element is a prefix which is followed by one or more suffixes. SageMaker appends the suffix elements to the prefix to get a full set of S3Uri. Note that the prefix must be a valid non-empty S3Uri that precludes users from specifying a manifest whose individual S3Uri is sourced from different S3 buckets.
The following code example shows a valid manifest format:
```
[ {"prefix": "s3://customer_bucket/some/prefix/"},
```
```
 "relative/path/to/custdata-1",
```
```
 "relative/path/custdata-2",
```
```
 ...
```
```
 "relative/path/custdata-N"
```
```
]
```
This JSON is equivalent to the following S3Uri list:
```
s3://customer_bucket/some/prefix/relative/path/to/custdata-1
```
```
s3://customer_bucket/some/prefix/relative/path/custdata-2
```
```
...
```
```
s3://customer_bucket/some/prefix/relative/path/custdata-N
```
The complete set of S3Uri in this manifest is the input data for the channel for this data source. The object that each S3Uri points to must be readable by the IAM role that SageMaker uses to perform tasks on your behalf.

s3DataSource_attributeNames :: Lens' S3DataSource (Maybe [Text]) Source #

A list of one or more attribute names to use that are found in a specified augmented manifest file.

s3DataSource_instanceGroupNames :: Lens' S3DataSource (Maybe [Text]) Source #

A list of names of instance groups that get data from the S3 data source.

s3DataSource_s3DataDistributionType :: Lens' S3DataSource (Maybe S3DataDistribution) Source #

If you want SageMaker to replicate the entire dataset on each ML compute instance that is launched for model training, specify FullyReplicated.

If you want SageMaker to replicate a subset of data on each ML compute instance that is launched for model training, specify ShardedByS3Key. If there are n ML compute instances launched for a training job, each instance gets approximately 1/n of the number of S3 objects. In this case, model training on each machine uses only the subset of training data.

Don't choose more ML compute instances for training than available S3 objects. If you do, some nodes won't get any data and you will pay for nodes that aren't getting any training data. This applies in both File and Pipe modes. Keep this in mind when developing algorithms.

In distributed training, where you use multiple ML compute EC2 instances, you might choose ShardedByS3Key. If the algorithm requires copying training data to the ML storage volume (when TrainingInputMode is set to File), this copies 1/n of the number of objects.

s3DataSource_s3DataType :: Lens' S3DataSource S3DataType Source #

If you choose S3Prefix, S3Uri identifies a key name prefix. SageMaker uses all objects that match the specified key name prefix for model training.

If you choose ManifestFile, S3Uri identifies an object that is a manifest file containing a list of object keys that you want SageMaker to use for model training.

If you choose AugmentedManifestFile, S3Uri identifies an object that is an augmented manifest file in JSON lines format. This file contains the data you want to use for model training. AugmentedManifestFile can only be used if the Channel's input mode is Pipe.

s3DataSource_s3Uri :: Lens' S3DataSource Text Source #

Depending on the value specified for the S3DataType, identifies either a key name prefix or a manifest. For example:

A key name prefix might look like this: s3://bucketname/exampleprefix
A manifest might look like this: s3://bucketname/example.manifest
A manifest is an S3 object which is a JSON file consisting of an array of elements. The first element is a prefix which is followed by one or more suffixes. SageMaker appends the suffix elements to the prefix to get a full set of S3Uri. Note that the prefix must be a valid non-empty S3Uri that precludes users from specifying a manifest whose individual S3Uri is sourced from different S3 buckets.
The following code example shows a valid manifest format:
```
[ {"prefix": "s3://customer_bucket/some/prefix/"},
```
```
 "relative/path/to/custdata-1",
```
```
 "relative/path/custdata-2",
```
```
 ...
```
```
 "relative/path/custdata-N"
```
```
]
```
This JSON is equivalent to the following S3Uri list:
```
s3://customer_bucket/some/prefix/relative/path/to/custdata-1
```
```
s3://customer_bucket/some/prefix/relative/path/custdata-2
```
```
...
```
```
s3://customer_bucket/some/prefix/relative/path/custdata-N
```
The complete set of S3Uri in this manifest is the input data for the channel for this data source. The object that each S3Uri points to must be readable by the IAM role that SageMaker uses to perform tasks on your behalf.