1. AWS Identity and Access Management (IAM) Policy and User

1. AWS Identity and Access Management (IAM) Policy and User

Dharamjeet Singh's photo
Dharamjeet Singh
·

10 min read

1. AWS Identity and Access Management (IAM) Policy and User

Steps:

  1. Create an IAM User:

    • Go to the AWS Management Console > IAM.

    • Navigate to "Users" > "Add Users."

    • Enter a username and select the "Access Type" (Programmatic access or AWS Management Console access).

    • Attach policies directly or add the user to a group with predefined policies.

Example Policy for Full Access to S3:

    {
      "Version": "2012-10-17",
      "Statement": [
        {
          "Effect": "Allow",
          "Action": "s3:*",
          "Resource": "*"
        }
      ]
    }

  1. Attach an IAM Policy to a User or Group:

    • Navigate to "Policies" > "Create Policy."

    • Use the visual editor or JSON to define the policy.

    • Attach the policy to the IAM user/group.

  2. Test Access:

    • Log in with the IAM user credentials.

    • Attempt the actions defined in the policy.

    • 1. AWS Identity and Access Management (IAM) (Continued)

      Grant Cross-Account Access:

      1. Create a Role for Cross-Account Access:

        • In the source account, navigate to "Roles" > "Create Role."

        • Choose "Another AWS Account" and provide the target account ID.

        • Attach a policy for the actions and resources you want to grant.

Example Policy for Cross-Account Access to S3:

            {
              "Version": "2012-10-17",
              "Statement": [
                {
                  "Effect": "Allow",
                  "Action": "s3:*",
                  "Resource": "arn:aws:s3:::example-bucket/*"
                }
              ]
            }

  1. Switch Role in the Target Account:

    • Log in to the target account, and use the “Switch Role” feature with the source account ID and role name.

  2. Test the Access:

    • Verify by accessing the specified resources.

2. Amazon Simple Storage Service (Amazon S3) Bucket

Steps:

  1. Create an S3 Bucket:

    • Go to the AWS Management Console > S3.

    • Click "Create Bucket."

    • Enter the bucket name and region.

  2. Apply Permissions:

    • Use Bucket Policies to control access.

    • Attach an IAM role with the necessary S3 permissions.

Example Bucket Policy to Allow Public Read:

    {
      "Version": "2012-10-17",
      "Statement": [
        {
          "Effect": "Allow",
          "Principal": "*",
          "Action": "s3:GetObject",
          "Resource": "arn:aws:s3:::example-bucket/*"
        }
      ]
    }

  1. Upload and Manage Data:

    • Use the S3 console, CLI, or SDKs to upload objects.

    • Configure lifecycle policies for data management.

    • 2. Amazon Simple Storage Service (Amazon S3) Bucket (Continued)

      Set Up Versioning and Replication:

      1. Enable Versioning:

        • In the S3 bucket settings, enable versioning to keep multiple versions of objects.

      2. Set Up Cross-Region Replication:

        • Enable replication to copy objects automatically to a different region.

        • Specify the destination bucket and attach an IAM role.

Example IAM Role for Replication:

            {
              "Version": "2012-10-17",
              "Statement": [
                {
                  "Effect": "Allow",
                  "Action": "s3:ReplicateObject",
                  "Resource": "arn:aws:s3:::destination-bucket/*"
                }
              ]
            }

  1. Test Replication:

    • Upload objects to the source bucket and verify their presence in the destination bucket.

3. AWS Lambda Function

Steps:

  1. Create a Lambda Function:

    • Go to the AWS Lambda console.

    • Click "Create Function" and select the runtime (e.g., Python, Node.js).

    • Define the function name and IAM role.

  2. Write and Deploy the Code:

    • Write code directly in the inline editor or upload a .zip package.

    • Deploy the function.

Example: Python Code to Process S3 Events:

    import json

    def lambda_handler(event, context):
        print("Event Received:", json.dumps(event))
        return {"statusCode": 200, "body": "Success"}

  1. Test the Function:

    • Configure a test event in the Lambda console.

    • Trigger the Lambda function.

  2. Integrate with Other AWS Services:

    • Example: Trigger Lambda from S3 or API Gateway.

    • 3. AWS Lambda Function (Continued)

      Schedule a Lambda Function:

      1. Create an EventBridge Rule:

        • Go to the EventBridge console and create a rule.

        • Choose a schedule expression (e.g., cron or rate).

      2. Link the Rule to the Lambda Function:

        • Select the Lambda function to trigger.

        • Test the function execution.

Example: Scheduled Cleanup Function:

            import boto3

            def lambda_handler(event, context):
                s3 = boto3.client('s3')
                response = s3.list_objects_v2(Bucket='example-bucket')
                for obj in response.get('Contents', []):
                    if obj['LastModified'].date() < (datetime.now().date() - timedelta(days=30)):
                        s3.delete_object(Bucket='example-bucket', Key=obj['Key'])

4. Amazon Kinesis Data Firehose Delivery Stream

Steps:

  1. Create a Delivery Stream:

    • Go to the Kinesis Data Firehose console.

    • Click "Create Delivery Stream."

    • Select a destination (e.g., S3, Redshift, Elasticsearch).

  2. Configure Transformations:

    • Optionally use Lambda for data transformations.

    • Attach an IAM role to grant permissions.

  3. Send Data:

    • Use the Kinesis Agent or SDK to send data to the Firehose delivery stream.

Example: Python SDK to Send Data:

    import boto3

    firehose = boto3.client('firehose')
    response = firehose.put_record(
        DeliveryStreamName='example-stream',
        Record={'Data': 'example-data'}
    )

  1. Monitor Delivery:

    • Use CloudWatch to monitor and debug issues.

    • 4. Amazon Kinesis Data Firehose Delivery Stream (Continued)

      Enable Compression and Encryption:

      1. Compression:

        • Choose a compression format (e.g., GZIP) during delivery stream setup.

      2. Encryption:

        • Enable server-side encryption using AWS KMS.

      3. Monitor Data Flow:

        • Use CloudWatch metrics like DeliveryToS3.Success to ensure reliable delivery.

Example Use Case:

  • Compress and encrypt log data before storing it in S3.

5. Amazon API Gateway

Steps:

  1. Create an API:

    • Go to the API Gateway console.

    • Click "Create API" and choose REST or HTTP API.

  2. Define Resources and Methods:

    • Add resources (e.g., /example) and methods (e.g., GET, POST).

  3. Integrate with Lambda:

    • Link API Gateway methods to a Lambda function.

  4. Deploy the API:

    • Create a stage (e.g., dev, prod).

    • Test the API using the stage URL.

    • 5. Amazon API Gateway (Continued)

      Enable Caching:

      1. Set Up a Cache:

        • Navigate to API Gateway > Stage > Enable Caching.

        • Specify the cache size.

      2. Test Cached Responses:

        • Verify that repeated requests to the same resource return cached results.

      3. Secure the API:

        • Use API keys or IAM authorization to restrict access.

Example Usage:

  • Cache frequently accessed product details to reduce backend calls.

6. Amazon Athena Table

Steps:

  1. Create an S3 Bucket for Query Results:

    • Set up an S3 bucket to store query results.

  2. Define a Table:

    • Use the Athena console to define a table schema based on data in S3.

Example Table Definition Query:

    sql
    CREATE EXTERNAL TABLE IF NOT EXISTS logs (
        id STRING,
        event_time STRING,
        event_type STRING
    )
    STORED AS PARQUET
    LOCATION 's3://example-bucket/logs/';

  1. Run Queries:

    • Use the Athena query editor to execute SQL queries on the defined table.

    • 6. Amazon Athena Table (Continued)

      Partitioning Data for Optimization:

      1. Organize Data by Partitions:

        • Store data in folders based on date, region, etc.

Example Folder Structure:

            bash
            codes3://example-bucket/logs/year=2025/month=01/day=01/

  1. Create a Partitioned Table:

     sql
 CREATE EXTERNAL TABLE logs (
     id STRING,
     event_time STRING,
     event_type STRING
 )
 PARTITIONED BY (year STRING, month STRING, day STRING)
 STORED AS PARQUET
 LOCATION 's3://example-bucket/logs/';

  2. Add Partitions:

     sql
 ATER TABLE logs ADD PARTITION (year='2025', month='01', day='01') LOCATION 's3://example-bucket/logs/year=2025/month=01/day=01/';

7. Amazon QuickSight Dashboards

Steps:

  1. Create a Data Source:

    • Connect to S3, Athena, Redshift, or other data sources.

  2. Prepare Data:

    • Use QuickSight to clean and transform the data.

  3. Build Dashboards:

    • Use charts, graphs, and widgets to create interactive dashboards.

  4. Share and Publish:

    • Share dashboards with users or groups in your organization.

Example Dashboard Use Case:

  • Visualize web traffic trends from an Athena table.

  • 7. Amazon QuickSight Dashboards (Continued)

    Automate Dashboard Updates:

    1. Set Up Scheduled Refresh:

      • Configure automatic data refresh for your datasets.

    2. Embed Dashboards:

      • Use the QuickSight API to embed dashboards into web applications.

Example Embed Code (JavaScript):

             var embeddingOptions = {
                url: "https://quicksight.aws.amazon.com/embed/",
                container: document.getElementById("dashboardContainer"),
                height: "800px",
                width: "100%"
            };
            QuickSightEmbedding.embedDashboard(embeddingOptions);

  1. Monitor Dashboard Usage:

    • Use QuickSight's built-in metrics to track usage and performance.

Next Steps:

  • Experiment with combining services (e.g., S3 triggering Lambda, Athena querying S3 data, API Gateway invoking Lambda).

  • Implement monitoring and logging using CloudWatch for all services.

  • Optimize cost and security by reviewing configurations regularly.

  • 1. Integrating API Gateway, Lambda, S3, Athena, and QuickSight

    Use Case:

    Create a data processing pipeline where data is uploaded through an API, processed by a Lambda function, stored in S3, queried using Athena, and visualized in QuickSight.

    Step 1: API Gateway to Accept Data

    1. Create a REST API in API Gateway:

      • Go to the API Gateway console.

      • Click "Create API" > "REST API" > Build.

      • Name your API (e.g., DataProcessorAPI).

    2. Create a Resource and Method:

      • Add a resource (e.g., /data).

      • Add a POST method to accept data.

    3. Integrate with Lambda:

      • Set the integration type to "Lambda Function."

      • Select or create a Lambda function (details in the next step).

    4. Deploy the API:

      • Create a stage (e.g., dev) and deploy the API.

      • Test the endpoint using a tool like Postman or curl:

          bash
   codecurl -X POST -H "Content-Type: application/json" -d '{"key":"value"}' https://<api-id>.execute-api.<region>.amazonaws.com/dev/data

Step 2: Lambda to Process Data

  1. Create a Lambda Function:

    • Go to the Lambda console.

    • Click "Create Function" and select "Author from scratch."

    • Choose a runtime (e.g., Python 3.9).

  2. Write the Lambda Function:

     import json
 import boto3
 import uuid

 s3 = boto3.client('s3')

 def lambda_handler(event, context):
     # Extract data from API Gateway event
     data = json.loads(event['body'])
     unique_id = str(uuid.uuid4())
     bucket_name = 'your-s3-bucket'
     key = f'data/{unique_id}.json'

     # Save data to S3
     s3.put_object(Bucket=bucket_name, Key=key, Body=json.dumps(data))
     return {
         'statusCode': 200,
         'body': json.dumps({'message': 'Data saved successfully', 'id': unique_id})
     }

  3. Assign IAM Role:

    • Attach a role with S3 write permissions to the Lambda function.

    • Example Policy:

        {
    "Version": "2012-10-17",
    "Statement": [
      {
        "Effect": "Allow",
        "Action": "s3:PutObject",
        "Resource": "arn:aws:s3:::your-s3-bucket/*"
      }
    ]
  }

  4. Test Lambda Function:

    • Simulate an API Gateway event in the Lambda console to verify data is stored in S3.

Step 3: Storing and Organizing Data in S3

  1. Create an S3 Bucket:

    • Go to the S3 console and create a bucket (e.g., data-pipeline-bucket).

  2. Organize Data:

    • Store incoming data in folders for better query performance (e.g., data/2025/01/05/).

Step 4: Querying Data with Athena

  1. Set Up an Athena Table:

    • Open the Athena console.

    • Create a table pointing to the S3 location where data is stored.

Example Table Definition:

        sql
        CREATE EXTERNAL TABLE IF NOT EXISTS data_table (
            id STRING,
            data_field1 STRING,
            data_field2 STRING
        )
        STORED AS JSON
        LOCATION 's3://data-pipeline-bucket/data/';

  1. Run SQL Queries:

    • Use the Athena query editor to analyze the uploaded data:

        sql
  SELECT * FROM data_table WHERE data_field1 = 'value';

  2. Save Query Results:

    • Save the query results to an S3 bucket for use in QuickSight.

Step 5: Visualizing Data in QuickSight

  1. Create a Dataset in QuickSight:

    • Go to the QuickSight console.

    • Create a new dataset using Athena as the source.

    • Select the table created in Athena.

  2. Build a Dashboard:

    • Add visualizations like bar charts, pie charts, or line graphs.

    • Example: Show the count of records by data_field1.

  3. Share the Dashboard:

    • Share the dashboard with users in your organization or embed it in an application.

2. Monitoring and Logging

  1. Enable CloudWatch for API Gateway and Lambda:

    • View metrics such as invocation count, error rates, and latency.

    • Use CloudWatch Logs to debug issues.

  2. Set Up Alarms:

    • Configure alarms to notify when Lambda errors exceed a threshold.

3. Security and Access Control

  1. Use IAM Roles:

    • Assign minimal permissions to Lambda and API Gateway roles.

    • Example: Grant Lambda permission only to write to a specific S3 bucket.

  2. Enable S3 Bucket Policies:

    • Restrict access to the bucket:

        {
    "Version": "2012-10-17",
    "Statement": [
      {
        "Effect": "Deny",
        "Principal": "*",
        "Action": "s3:*",
        "Resource": "arn:aws:s3:::your-s3-bucket/*",
        "Condition": {
            "Bool": {
                "aws:SecureTransport": "false"
            }
        }
      }
    ]
  }

  3. Enable Logging and Encryption:

    • Turn on S3 server-side encryption and access logs.

4. Automation and Scalability

  1. Use Step Functions for Orchestration:

    • Orchestrate Lambda functions to handle complex workflows.

Example State Machine:

  • API Gateway > Lambda (Data Processing) > Lambda (Data Validation) > S3.

    1. Use Kinesis Firehose for Real-Time Data Streaming:

  • Replace the S3 integration with Kinesis Firehose to stream data directly to S3 or Redshift.

Final Workflow:

  1. API Gateway accepts data and triggers Lambda.

  2. Lambda processes the data and stores it in S3.

  3. Athena queries the data stored in S3.

  4. Results are visualized using QuickSight dashboards.

  5. Monitoring and logging are handled via CloudWatch.

By combining these services, you can create an end-to-end serverless data processing pipeline that is scalable, secure, and cost-effective. Let me know if you need further elaboration on any step!

1aws-identity-and-access-managementAWSfreecourseralambdaLambda ExpressionLambda functionIAMiam role in aws