AWS CloudTrail Started/Stopped

 CloudTrail is an essential feature for any running AWS account, it’s a record of each and every action that occurs. Without it, company Security would not be able to detect any potential harmful situations and would be left completely in the dark about activity in AWS. Steps from Amazon to enable this can be found here:

Creating a Trail in the Console

You can configure your trail for the following:

• Specify if you want the trail to apply to all regions or a single region.

• Specify an Amazon S3 bucket to receive log files.

• For management and data events, specify if you want to log read-only, write-only, or all events.

To create a CloudTrail trail with the AWS Management Console

1. Sign in to the AWS Management Console and open the CloudTrail console at https://console.aws.amazon.com/cloudtrail/.

2. Choose the region where you want the trail to be created.

3. Choose Get Started Now.

Tip

If you do not see Get Started Now, choose Trails, and then choose Create trail.

4. On the Create Trail page, for Trail name, type a name for your trail. For more information, see CloudTrail Trail Naming Requirements.

5. For Apply trail to all regions, choose Yes to receive log files from all regions. This is the default and recommended setting. If you choose No, the trail logs files only from the region in which you create the trail.

6. For Management events, for Read/Write events, choose if you want your trail to log All, Read-only, Write-only, or None, and then choose Save. By default, trails log all management events. For more information, see Management Events.

7. For Data events, you can specify logging data events for Amazon S3 buckets, for AWS Lambda functions, or both. By default, trails don't log data events. Additional charges apply for logging data events. For CloudTrail pricing, see AWS CloudTrail Pricing.

You can select the option to log all S3 buckets and Lambda functions, or you can specify individual buckets or functions.

For Amazon S3 buckets:

• Choose the S3 tab.

• To specify a bucket, choose Add S3 bucket. Type the S3 bucket name and prefix (optional) for which you want to log data events. For each bucket, specify whether you want to log Read events, such as GetObject, Write events, such as PutObject, or both. For more information, see Data Events.

• To log data events for all S3 buckets in your AWS account, select Select all S3 buckets in your account. Then choose whether you want to log Read events, such as GetObject, Write events, such as PutObject, or both. This setting takes precedence over individual settings you configure for individual buckets. For example, if you specify logging Read events for all S3 buckets, and then choose to add a specific bucket for data event logging, Read is already selected for the bucket you added. You cannot clear the selection. You can only configure the option for Write.

Note

Selecting the Select all S3 buckets in your account option enables data event logging for all buckets currently in your AWS account and any buckets you create after you finish creating the trail. It also enables logging of data event activity performed by any user or role in your AWS account, even if that activity is performed on a bucket that belongs to another AWS account.

If the trail applies only to one region, selecting the Select all S3 buckets in your account option enables data event logging for all buckets in the same region as your trail and any buckets you create later in that region. It will not log data events for Amazon S3 buckets in other regions in your AWS account.

For Lambda functions:

• Choose the Lambda tab.

• To specify logging individual functions, select them from the list.

Note

If you have more than 15,000 Lambda functions in your account, you cannot view or select all functions in the CloudTrail console when creating a trail. You can still select the option to log all functions, even if they are not displayed. If you want to log data events for specific functions, you can manually add a function if you know its ARN. You can also finish creating the trail in the console, and then use the AWS CLI and the put-event-selectors command to configure data event logging for specific Lambda functions. For more information, see Managing Trails.

• To log data events for all Lambda functions in your AWS account, select Log all current and future functions. This setting takes precedence over individual settings you configure for individual functions. All functions are logged, even if all functions are not displayed.

Note

If you are creating a trail for all regions, this selection enables data event logging for all functions currently in your AWS account, and any Lambda functions you might create in any region after you finish creating the trail. If you are creating a trail for a single region, this selection enables data event logging for all functions currently in that region in your AWS account, and any Lambda functions you might create in that region after you finish creating the trail. It does not enable data event logging for Lambda functions created in other regions.

Logging data events for all functions also enables logging of data event activity performed by any user or role in your AWS account, even if that activity is performed on a function that belongs to another AWS account.

8. For Storage location, for Create a new S3 bucket, choose Yes to create a bucket. When you create a bucket, CloudTrail creates and applies the required bucket policies.

Note

If you chose No, choose an existing S3 bucket. The bucket policy must grant CloudTrail permission to write to it. For information about manually editing the bucket policy, see Amazon S3 Bucket Policy for CloudTrail.

9. For S3 bucket, type a name for the bucket you want to designate for log file storage. The name must be globally unique. For more information, see Amazon S3 Bucket Naming Requirements.

10. To configure advanced settings, see Configuring Advanced Settings for Your Trail. Otherwise, choose Create.

11. The new trail appears on the Trails page. The Trails page shows the trails in your account from all regions. In about 15 minutes, CloudTrail publishes log files that show the AWS API calls made in your account. You can see the log files in the S3 bucket that you specified.

Note

You can't rename a trail after it has been created. Instead, you can delete the trail and create a new one.

 https://docs.aws.amazon.com/awscloudtrail/latest/userguide/cloudtrail-create-and-update-a-trail.html

AWS security group secure setup

Security groups are an essential control when protecting both a resource or network in the Amazon Web Services cloud. They are rules that dictate who, entities by IP address and protocol, can communicate with your hosted networks and/or individual devices. A misconfigured rule can lead to near fatal results, granting access to the entire internet. With open access, malicious attackers will certainly make attempts to access the resource, increasing the chances of compromise by a wide margin.

Permitting traffic from any internet host to any port is generally discouraged, as a security best practice. Rules configured this way expose all ports and services on a host to any internet IP. This can result in malicious actors scanning administrative ports like SSH or RDP and attempting to brute force entry. These rules can almost always be reconfigured to be much more restrictive while still allowing legitimate users to connect to your application.

There are situations where it would be hard to avoid an any rule, for example, a web site having the necessity of being open to the world on ports 80 and 443. Very few situations, if none at all, should allow as a source all clients. Access needs to be explicitly stated where it is originating from. To make the process easier, IP ranges are commonly used to avoid having to add a running list host by host. For example, all of company xyz networks can be added as a source with a statement such as 128.59.0.0/16

An example any/any Security Group rule is shown below. Note the 'All' under 'Protocol' and 'Port Range'. 'All' in these columns means the remote host can connect to ANY port/service on your VM. Note also the 'Source' column. '0.0.0.0/0' and '::/0' means ANY IP address.

Instead, it is recommended to open traffic to ONLY the ports and IP ranges that your application requires. This may mean creating more complex rules, but it will result in greatly improved security. In the following example, the Security Group is configured to permit traffic from any IP address, but ONLY to ports 80 and 443, for web traffic. With this configuration, administrative ports like SSH and RDP are not open to the internet.

refer: https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/using-network-security.html

Disable HTTP OPTIONS Method in Apache

 In RHEL installations the httpd.conf file located in /etc/httpd/ you will need to enable mod_rewrite:

Uncomment or add:

LoadModule rewrite_module /usr/lib/apache2/modules/mod_rewrite.so

For each virtual host these directives need to be set in order to filter out OPTIONS method:

RewriteEngineOn

RewriteCond%{REQUEST_METHOD} ^OPTIONS

RewriteRule .* - [F

Key Vault Keys Should Have an Expiration Date

 Cryptographic keys should have a defined expiration date and not be permanent. Keys that are valid forever provide a potential attacker with more time to compromise the key. It is a recommended security practice to set expiration dates on cryptographic keys.

To enable this:

To enable an expiration date on your key:
1. Log in to the Azure portal and select your key vault.
2. Open the 'Keys' tab.
3. Find all keys in the table that do not have an expiration date.
4. Select a key.
5. Select the current version of the key.
6. Select the box corresponding to 'Set expiration date'.
7. Select 'Save'.

Email Notification to Subscription Owner for High Severity Alerts Should be Enabled

Description:

To ensure your subscription owners are notified when there is a potential security breach in their subscription, set email notifications to subscription owners for high severity alerts in Security Center.

 

Solution: 

To configure email notifications:
1. From Security Center's menu, select Pricing & settings.
2. Select the relevant subscription.
3. Select 'Email notifications'.
4. In the email recipient section, select users with 'Owner' role.
5. In the 'Notification type' area, ensure mails are sent regarding security alerts from severity 'high'.
6. Select 'Save'.

Container Registries Should Not Allow Unrestricted Network Access

Description:

Azure container registries by default accept connections over the internet from hosts on any network. To protect your registries from potential threats, allow access from only specific public IP addresses or address ranges. If your registry doesn't have an IP/firewall rule or a configured virtual network, it will appear in the unhealthy resources. 

 

Solution/Reference: 

To enable VNet/Firewall rules for a registry:
1. In the Azure Portal, navigate to your registry in the portal
2. Under Networking settings, on the Public access tab, select allow public access from 'Selected networks' instead of 'All Networks'
3. Under Firewall, enter a public IP address, such as the public IP address of a VM in a virtual network. Or, enter an address range in CIDR notation that contains the VM's IP address
4. Select save. 

Learn more about Container Registry network rules here: https://aka.ms/acr/portal/public-network and here https://aka.ms/acr/vnet.

For more information, see: https://aka.ms/acr/portal/public-network and https://aka.ms/acr/vnet.

Secure Coding Practices

 

Handling Passwords and Secret Keys using Environment Variables

Whether one is programming in a development environment or migrating code to a production environment, a major security risk can be posed when using plaintext for keys, passwords, and secrets in code. When credentials are leaked, attackers can use that to leverage access into systems, environments, and applications which they could use to gain even more access and privileges in these systems. The generally accepted best practice way to remediate this risk is to use environment variables in almost every situation.

Environment variables are a set of dynamic named values, stored within the system that are used by applications launched in shells or subshells. Ultimately, an environment variable is a variable with a name and an associated value that can not be changed within code. It is system specific. 

How to Implement Environment Variables

In a *nix system, the command is: 

export PASSWORD="password”

In a Windows system, the command is:

 set PASSWORD="password”

Referencing the environment variable in different programming languages:

Python:

os.environ.get(PASSWORD)

JavaScript:

 process.env.PASSWORD

C#:

char * secure_getenv (const char *PASSWORD)

C++:

std::getenv("PASSWORD")

Java:

System.getenv(“PASSWORD”)

Shell/Bash:

$PASSWORD

 These methods are all security best practice and are required in development, test, and production environments.

Remove/Approve Untrusted Boot Components

Description:

With Secure Boot enabled, all OS boot components (boot loader, kernel, kernel drivers) must be signed by trusted publishers. Security Center has identified untrusted OS boot components on one or more of your Linux machines. To protect your machines from potentially malicious components, add them to your allow list or remove the identified components.

Remediation:

Investigate the untrusted boot components. If they are legitimate, add them to the allow list. Otherwise, remove them.

Reference:

More detail on the concept of Secure Boot can be found here: https://docs.microsoft.com/en-us/windows-hardware/design/device-experiences/oem-secure-boot

Sensitive Data in Your SQL Databases Should Be Classified

Description:

Azure SQL DB Data discovery & classification provides capabilities for discovering, classifying, labeling, and protecting the sensitive data in your databases. Once your data is classified, you can use Azure SQL DB auditing to audit access and monitor the sensitive data. Azure SQL DB also enables Advanced Threat Protection features which creates intelligent alerts based on changes in the access patterns to the sensitive data.

Remediation:

To remediate this recommendation:
1. In your SQL database, go to 'Advanced Data Security' and click 'Data Discovery and Classification'.
2. Review the recommended classifications.
3. Apply the relevant recommendations and dismiss the ones that are not applicable.
4. Please note that the updated health status for the database will not be reflected immediately and can take up to a week to refresh. You can make this happen faster by triggering a database Vulnerability Assessment scan: in your SQL database go to 'Advanced Data Security', click 'Vulnerability Assessment' and click 'Scan'. The health status of the database will be updated within 1 day from scan completion.

Openstack private cloud, a shift in cloud strategy?

 

OpenStack is an open source platform that uses pooled virtual resources to build and manage private and public clouds. While i knew about it , but never really knew what is the use case for it with so many cloud providers around. Who would want private cloud, private cloud is dead per many cloud vendors. But the reality of cloud costing is hitting the enterprises hard, and i expect will hit even more harder if they continue to ignore it.

OpenStack is a combination of open source tools (also known as projects) that use pooled virtual resources to build and manage private, hybrid and public clouds, all of which can be highly customized per customer’s business needs. OpenStack is one of the leading open-source options for building the private cloud environment. Though enterprises lean towards cloud adoption, they are realizing that having a private cloud addresses many security concerns. As per an open stack survey, the deployment of openstack is going up year after year.

It seemed to be that openstack looked similar to virtualization. But its not quite same. OpenStack is an open source private cloud platform designed to manage distributed compute, network and storage resources in the data center. In principle, OpenStack aggregates physical resources into one big pool and allocates virtual resources out of this pool to users who can request them on-demand through a self-service portal or application programming interfaces (APIs). They use regular virtualization tools to enable cloud native capabilities.

One might think that OpenStack has the potential to do with the cloud market the same thing that Linux did to commercial Unix market, effectively eliminating players such as Sun, HP-UX, AIX. Unfortunately for OpenStack this analogy is only partially true. It does provide a free and open source software alternative for the commercial cloud. “Software only” is the key here. Yes, it runs on commodity hardware, but that’s not what makes cloud attractive. Getting rid of all the datacenter -management burden, pay for only what you use, the economy of scale which make public cloud attractive still hold true. Any strategy is business driven and this still holds true . So, while we may not see major impact on public cloud. We might see enterprises evaluating the dev and staging systems using Openstack as its cost effective over a long period of time and serves well from Capex and Opex perspective.

Cloud Governance best laid plans : Part 1

 I have been working on cloud from 7 years now . I started with OCIC Oracle and then OCI and now with Azure, AWS and GCP. When in operations, the focus was set it right, implement it well, check it before go live, keep it running with RTO/RPO and maintain SLA. But few years ago the focus was on good implementation practices. Good cloud governance is not just about technical changes , its also involves changes to your corporate policies. Corporate policies are always tied directly to the business level risks. With only good corporate policies we can think of healthy cloud governance.

Corporate policies specifically come under 3 sections:

Business risk

Risk drives policy, and it influences monitoring and enforcement requirements. Again the risk, which a small company has versus a fortune 500 company have might be different. Also, on premise to cloud the risks are very different. The scale of deployment, size of company, customer base etc change business risks across organizations. Business risk can be driven in MVP(minimum viable product) similar to product development. Go small and then expand as the deployment increases than look for 100% mapped governance policies. You can never wait for complete holistic governance, earlier the better even if small scale.

2. Policy and compliance

Converting risks decisions into policy statements and getting compliance is crucial. Review existing policies around cloud and create or update them as needed. Most cloud providers now publish the security policies for each product, most of it is baseline. You can tweak those based on your corporate policies.

3. Processes

After defining policies the next step is to define processes which support those policies. The processes will support policy adherence. This might also include establishing cloud governance team. Have regular check on policies and processes mapping to keep them current and aligned to business. There is vast need of training when we establish these processes to keep the employees informed. Define what path will be taken when there is a violation. Automation of these processes is a great way to mitigating risks, as we take data based decisions against just an assumption.

Will discuss the cloud governance in the next post

Log4j : Frenzy , but you want to know what is it all about?

On December 9th, 2021 a new vulnerability was discovered affecting the java logging library, Log4j. Since this news broke out, threat actors around the world have rushed to take advantage of this easy-to-exploit vulnerability and wide-scale attacks are now ongoing. All companies and its engineers, security, support teams are also busy in identifying and fixing this issue. Almost every enterprise company is impacted by it. It has base CVSS score of 10!! .

CVE-2021–44228 is a Remote Code Execution vulnerability in the Apache Log4j library, a Java-based logging tool widely used in applications around the world. This vulnerability allows an attacker who can control log messages to execute arbitrary code loaded from attacker-controlled servers — and I anticipate that most apps using the Log4j library will meet this condition.

This is really good video to understand for those who want to understand what all this is about.

https://www.youtube.com/watch?v=r4liwEQm14o

https://www.youtube.com/watch?v=OGIH3ulSk-Q&t=4s

Then there are resources which can used to check your own applications and code. Using tools like synk to look at the code. The presence of JAR files belonging to the log4j library can indicate an application is potentially susceptible to CVE-2021–44228. The specific files to search for should match the following following pattern: “log4j-core-*.jar”. For example, on Windows, if the file is located in C:\Program Files\ApplicationName\log4j-core-version.jar it indicates ApplicationName should be investigated. On Linux, the lsof utility can show which processes currently have the JAR file in use and can be run via the following syntax: “lsof /path/to/log4j-core-version.jar;”

Using tools like Qualys to look at server level. If your organization uses the log4j library, upgrade to log4j-2.1.50.rc2 immediately. You should also be sure that your Java instance is up-to-date. Then looking up vendor documentation for patches or potential fixes. A patch for CVE-2021–44228 has been released, but unfortunately, we’re at the mercy of many of our vendors to push updates that completely patch the vulnerability. The log4j package may be bundled in with software you use provided by any given vendor. In this scenario, unfortunately, the vendors themselves will need to push the security updates downstream. As you assess your own risk and threat model, please consider the components of the software you use and especially what may be publicly accessible.

Resource library: Putting up few resources i came across:

Software list which is affected

IOC feeds and threat reports

Technical explanation

Security advisories

What do you do ??

If you are in engineering, support, security i am assuming you are going to have a busy year end depending on how much your organization uses these. You can either patch this for apply a quick fix for mitigation of immediate risk. For patching Apache Log4j2 version 2.15.0 fixes this vulnerability
but Log4j 2.15.0 requires Java 8. Therefore, teams that use Java 7 will need to upgrade Java before upgrading Log4j.

Upgrade takes planning and getting downtimes. So for immediate reduction in risk, you can do it at configuration level possible only for ≥ 2.10 only. You can set the system property log4j2.formatMsgNoLookups or the environment variable LOG4J_FORMAT_MSG_NO_LOOKUPS to true. For lower versions of 2 like 2.0 to 2.10.0, you have to remove the jndi lookup class from the classpath. Even version 1.0 is impacted, but since is out of support, nobody will mention explicit instructions, but you are at risk.

Ciao and good luck everyone, its a race against time !

Sane security practices

For the longest time IT systems have remained secure in datacenters ,with secure network devices, physical security and firewalls. I have worked in datacenter in the era where cloud was just starting up and nobody really seemed interested in cloud those days. The access to these systems was either from within LAN network and sometimes even had to work from server console inside the really cold server room. Imagining those days in the era of covid makes me wonder, what would we have done if we had no cloud. Imagine the disruption caused by that. Its unimaginable the scale of shutdown it would have caused, as everything from hospitals to insurance providers to transport to pharma depends on these.

The new buzz word all around is cybersecurity. In the past year most cloud providers like Azure, AWS, GCP ,Oracle etc have released and invested massively on security. From releasing features for compliance, to building dashboard for a clear view. Its been a huge improvement overall. From releasing blueprints which help build secure systems, to policies and tools to identify issues in the systems , to actually notify in case of real threats. While these tools existed in past , never were they as robust as they are now.

One consistent issue with most projects is that there is very less focus on security at early stages. I am not trying to generalize , but based on my observation in last few years, The focus is to lift and shift or just keep it running as how it was on private cloud. Public cloud is whole new ball game, there is identity security which is paramount. We all know least privilege's have to be assigned but that again is not focus area in early stages . Lets just install a server and just move all data, if its open to the internet then lets focus on that later. Even having worked with some vendor, i realized their sole focus was to spin up resources and just get it working. There were even few cases where pen testing reports had test data and not really been tested and these are tools which were selling saas security products. Imagine using them to secure your systems. This is true story btw , though i leave it upto the reader to discover this someday when there is a breach.

This gets you to question on how to build system designs ? I have some interest in system design after watching some interesting videos. When i try looking up few case studies, most people never focused on security in system design stage . It seemed like something to add on later after you have everything up and running. Then how do we build secure systems? How do our developers build security in systems than add security to systems??

Start early:

As i stated earlier, system design needs to have security at the early stages , as its easier and less cost intensive in future if systems are built securely . Again , with recent shift to move everything asap to cloud and lets build cloud native systems later, i see systems which are out there dangling insecurely on the cloud. There are tools which makes this simpler at early stages, than breaking systems to secure later.

Focused approach:

So, if you did not or could not focus at early stage then we get to a stage where it needs to be done post go live or even at early dev stages. With so many cloud native tools, vendor tools out there it difficult to choose. Many a times there are cloud native tools with no additional cost, there are even open sources which help do this. So identify the one which fulfils yours needs. Break it down per cloud provider, identify technology which works for all workloads and clouds. There is a lot of duplication in tooling which makes security expensive, which again leads to not so great “cost based security” decisions.

Too much focus on methods:

There seem to be a rush all around to adopt all types of methodologies which might not work for security practices. There needs a clear consideration on what works and what does not. We sometimes need to tweak process to save time and cost. The strict adhering to processes which just implicates on time makes security complicated. In turn it creates an all round culture of bypassing secure processes as the process to get secure itself is more complicated. Too much paper work, too many meetings, too many chaotic tools, too many reports all make it impracticable.

Decouple complicated processes:

Key to getting things in line is to make fixing things easier. Have a step by step approach to security and engineering working side by side. Most times the mind set shift is needed, where we need to build mindset at base level. A VP cannot fix a server or database open to internet. While he/she can make policies around it, there still is that engineer who need to understand the implication of such a violation. Make it matter to everyone, make a criteria for team success. Else, it will continue to be difficult to manage process.