Securing Success: Crafting Environments for Development, Innovation, and Security.

Introduction

This article is written for No-Agenda.Tech. That means you get the honest assessment with no agenda regarding products being sold, no agenda to technology being evangelized, and no agenda regarding companies being pushed.

Crafting fortified production environments begins with the network but only succeeds with a secure software development team and architecture. In today’s tech world, there’s a big disconnect. Security experts often lack insight into software development, while developers aren’t always focused on security. This mismatch leaves businesses open to avoidable threats that could be easily tackled without adding extra strain on teams. A recent Gartner report states that only 30% of product development teams use DevSecOps as of 2022 (Betts, 2023). With limited DevSecOps based development teams and security teams that don’t fully understand software development, the risk to business owners is high that a negative event will occur. This article will discuss solid ways to secure the network through proven practices as well as the steps needed to be addressed with software development.

Examples and stories help the understanding process, so throughout this article we will view each situation through the eyes of Tony’s Automotive Parts (TAP). TAP is not a real company and Tony, the CEO, is not a real person, but we can live and learn vicariously through Tony as he journeys through decisions regarding his company’s operation. Searching back over past No-Agenda articles, TAP has grown from a small business into a regional force in the automotive parts market. TAP has 20 retail stores and an e-commerce online system. The e-commerce solution is deployed and run as part of the parts distribution center. Tony is always looking to expand his business. He has decided his next expansion will be to create a real-time engine monitoring and tuning system that car enthusiasts can plug into their cars. Tony can OEM the modules, program them with his team’s software, and users can use the TAP developed mobile and web applications to adjust settings on their cars in real-time. This article will take us on the journey as TAP builds and deploys the system, they call TRASE, which stands for Tony’s Real-time

Common Frame of Reference

To understand TRASE as it’s being developed and used, we need to clarify some key terms and concepts. This section will explain these terms and provide a common foundation for our discussion. Think of it this way: Tony understands that creating a network that follows NIST SP-800 standards is like building a super secure fortress. This is a top industry practice that ensures strong protection against cyber threats. (NIST Special Publication 800-53. Revision 5. Security and Privacy Controls for Information Systems and Organizations, 2020). This standard and specifically, NIST SP-800.53 or NIST SP-800.171 are required for many government and high-end commercial environments (e.g., credit card processing systems) where applied security principles are required.

Production Environment

To contextualize NIST within TAP, it is good to define what is meant by a production environment. The production environment is computer and network systems where revenue is generated. From the perspective of TAP there are three parts to production, the retail store systems, the e-commerce at the distribution center systems, and the new TRASE system. While the Online store would also be considered part of this production system, our focus will just be on the TRASE system. The customer’s plugin module is not part of the TAP production system because the customer owns the module and has control over it.

Development Environment

To define what is meant by a development and test environment, from the perspective of TAP, the development environment is considered part of the non-production group of subnetworks where development, quality assurance, and pre-production testing are conducted. In the development subnetwork, developers are afforded the freedom to experiment with new software and hardware. The new components may or may not have been approved or commissioned for production environments.

Separation of Control

Much of the NIST SP-800 standard is based on the principles of controlled access, limited access, separation of control, and auditability. Separation of control can be defined for TAP by having separate roles and responsibilities at key points in the deployment and operations process. The development process is executed by Development Engineering while the operation of production systems is executed by Operations Engineering.

TAP Applies NIST for TRASE

After reading the 495-page NIST SP-800.53 standard, Tony has decided to provide some primary guidance for the teams to build the production environment. The guidance is depicted graphically in Figure 2 - Production Network Traffic Flow shown below and summarized in the bullets.

· Limit Access: Only a few TAP employees should have access to the production systems.

· Reduce Resources: Use the fewest possible computing resources for the production system.

· Control Network Connections: All network connections should start from within the production network and be tightly controlled.

· Secure Traffic: Carefully control traffic between production and non-production networks by specifying IP addresses, ports, and protocols.

· Restrict Storage: Only non-executable data should be stored on shared network storage devices, and access should be strictly controlled.

· No Internet Browsing: Production systems should not have general internet access to prevent potential security risks.

To keep our systems secure, we need to regularly scan for viruses and update our software. The best way to do this is by using a special piece of software, called an agent, in our production environment. This agent reaches out through the corporate network for virus definition updates and software patches to be deployed. This is not always practical. Where an agent is not available, strong restrictive firewall rules should be configured to create a pinhole connection initiated from corporate. The pinhole connection should consist of source and destination IP address filtering as well as protocol limitations.

The TAP operations team plans to push logs to a shared drive folder where they have Read/ Write (RD/WR) permissions and the engineering teams just has Read-Only (RO) permissions. Since software releases will be passed via a different mechanism and no data needs to be pushed from the development or corporate environments to the production environment, no folders are shared between development (corporate) and the operations team’s production network. In an ideal solution, the production network utilizes a controlled program to automatically initiate and transfer data to the access-controlled network share. In this manner, all activities can be monitored and audited on a regular basis.

Development Environment

The TAP software development director is asking the company to stand up a development and test network to be compliant with NIST and meet the needs of the development team. Think of this network like a safe playground where developers can experiment without affecting the main company network. The basic concepts of the network design are shown below in Figure 3 - Development and Test Network Flow. Network traffic for the development subnetwork must be tightly controlled. No traffic or network connections should be initiated from within the development subnetwork outward to the corporate network. Since traffic can flow into the developer network, developer laptop/computers can reside either within the developer network or on the corporate network. Other development assets such as DevSecOps solutions should reside inside the developer network. Computer scanning and patching rules inside the developer network must be applied in a custom, limited fashion. The developer network is used to replicate issues in production and will likely need to use very specific versions of software, many with known vulnerability issues. The purpose for creating the development laboratory type network is to allow the experimentation while protecting the larger corporate network as whole. In some cases, internet access may be required from the Development subnetwork. This connectivity should not permit access to other corporate assets and should maintain standard corporate filtering and deep packet inspection procedures.

Software Releases and Updates

Much of the NIST SP-800 standard is based on the principles of controlled access, limited access, separation of control, and auditability. Separation of control can be defined by having separate roles and responsibilities at key points in the deployment and operations process.

As shown in the figure above, Development Engineering only has access to the development and test environments. The Operations Engineering team only has access to the production environment. The interface for pushing and pulling executable data across the production/no-production barrier should be a repository device such as Nexus, Artifactory, or Azure Registry. Using an artifact repository provides controls on who can upload and download executables. They also prevent the direct execution of such code where a network share does not. Most of these types of registry repository devices also support versioning and automatic pruning which helps to prevent usage of the wrong version and minimized the utilized disk space for overall storage.

Summarize the network structure concepts

Designing and building a network for large-scale usage is a daunting task. Following the simplified guidance above and NIST SP-800 control requirements, TAP is constructing the network with production in mind. New code for the web solution and module software is being published by the development team into the artifact repository where it is pulled into production by the TAP operations team. TAP uses their Infrastructure as Code (IaC) scripts to install and update the infrastructure and software deployed. The Terraform scripts and Ansible playbooks are the same ones developed and tested by the development engineering team and proven in the QA environment before being published to the artifact repository. When log data and other issue artifacts are needed by the TAP development engineering team, the operations team uses a transfer program to send the data to the appropriate network shared resource. The TAP operations team monitors the production environment for proactive intervention. By following these design guidelines, TAP has constructed a network that minimizes vulnerabilities while taking advantage of the latest lean/agile DevSecOps principles. Now, let’s see how TAP’s development engineering team is creating, testing, and deploying the software needed to run TRASE.

DevSecOps and Software Development

The TRASE development team for TAP is responsible for all development, test, and deployment for the TRASE product line. This includes the responsible use of open-source code, writing quality code for the module and website, testing the module and website code, and creating the deployment solution for the QA and Production environments.

An eye on secure design

As one of the largest and newest initiatives, Tony, as CEO, has taken a distinct interest in the development of TRASE. After reading a Gartner article titled “3 Essential Steps to Enable Security in DevOps,” Tony has decided to hire a Security coach as recommended by the article (Betts, 2023). Tony describes the security coach to the development as part developer, part systems engineer, part security analyst, and part network engineer.

With the TAP Security Coach, John, as an integral part of the development team, they feel like they are well on their way to making a good and secure product. The TAP Security Coach is proposing to begin with a shift-left approach to automated security governance. John used Figure 6 - Shifting Security Thinking Left to describe what he meant by the term. The team starts out by establishing a build, deploy, and test DevSecOps pipeline. The pipeline will initially include Software Composition Analysis (SCA) and Static Application Security Testing (SAST) tools and later include Dynamic Application Security Testing (DAST). The team feels like DAST is a bridge too far until they have a better understanding of the deployed environment and something to put in it.

Software Composition Analysis (SCA)

The TAP Security Coach is having to answer questions and teach along the way to creating the DevSecOps pipeline. The first question John addresses is, what is SCA and why is it useful for the TRASE team. The following were some the reasons John described as the value of SCA.

• SCA as a Safety Gate: The best SCA tools generally act like a firewall for open-source incorporation by preventing the incorporation of code configured as not allowed. The system configurations allow for the inclusion or exclusion of various Open-Source Software (OSS) locations, licenses, and types of products to be incorporated. Some actually integrate directly into the developers integrated development environment to perform this action. Others, integrate only into the build system which is not as desirable but generally works adequately.

• License Compliance: SCA tools provide a mechanism to maintain the company’s license compliance. If the company has a policy of never using GPLv3, or some other license type, the SCA tools prevent them from being incorporated or prevent the build from continuing. The SCA tools also provides a log or report of every license used in a particular build of software. This is especially useful for OSS compliance when this data is included in a release report and provided to the customer to disclose the OSS used within the delivered product.

• Tracking Vulnerabilities: SCA tools provide Common Vulnerabilities and Exposures (CVE) lists for each OSS product incorporated in the build system. SCA vendors report exposures to a common database and many augment the CVE discovery with their own explorations. The National Vulnerability Database (NVD) and CERT/CC Vulnerability Notes Database provides common definitions and tracking of issues discovered and fixed (CVE.org, 2024). Most SCA tools allow for the configuration to halt the build process if a vulnerability is listed as too severe. These tools generate a report listing all of the CVEs for each product incorporated into a build. As with the licensing data in the report, the CVE data helps to determine and demonstrate the vulnerability of the software being produced.

Combining the elements of license control, OSS repository control, and vulnerability knowledge provides a powerful linage of the software supply chain of OSS being brought into the code base. The features of SCA help with the reliability of the software only when they are diligently maintained and viewed. John is directing the TRASE team to incorporate SCA into every build and the generate a software bill of materials (SBOM) for every release. The SBOM will contain a complete list of all of the OSS items, each license, the CVE value, and version of software being used in the code.

Static Application Security Testing (SAST)

John, the security coach has emphasized the importance of security with the TRASE team building a web enabled application (web server, webpage, and mobile application). As part of the pipeline buildup, John is insisting that SAST be included in the initial configuration. SAST provides analysis of code that is written by the TRASE development team as opposed to SCA which covers OSS. SAST helps:

• Understand Code Complexity: It shows how complex the code is and checks if it follows proper formatting.

• Find Common Errors: It looks for common security issues, like code injections, which are frequent in web applications.

Including SAST in the DevSecOps pipeline ensures that the code is secure and up to standard before it moves to the next stage. The SAST configuration decided on by the TRASE team shows them the complexity of their code, validates the formatting, and checks for all types of code injection failures common to web enabled applications.

IaC for automated deployment

The TRASE team has set up their environment so that components can be automatically deployed by dynamically creating the cloud resources needed to run them and deploying them after the resource is created. This is an important step in the DevSecOps solution because it enables automated functional testing to be conducted with every build.

Automated test for product reliability

The TAP Security Coach, John, is looking to have the team execute a test-driven-development methodology. That means tests are created as the functionality is created. TRASE uses different types of tests, such as:

• Functional Tests: Check if the features work as intended.

• Integration Tests: Ensure that different parts of the system work together.

• Use-Case Tests: Verify that the software handles specific scenarios correctly.

Automated testing, combined with automated deployment, helps catch issues early and ensures that the software remains reliable. Combining the automated deployment of components along with automated testing ensures the functionality created works and is regularly tested. Knowing immediately when some feature or function is broken makes fixing the issues much easier as the change set is much smaller regarding where the problem could exist. Having the developers write the tests in lieu of a test group also adds reliability to the product and prevents the inevitable finger pointing between groups. This methodology holds developers accountable for quality as they are responsible for the development from creation through deployment. It also allows developers to follow a Test-Driven-Development (TDD) methodology as part of their agile development process.

Generating a release report

With SCA processes in place in the DevSecOps environment, a Software Bill of Materials (SBOM) can be produced which shows the software supply chain linage. This provides visibility to what is contained in the software product and helps to catch vulnerabilities before they hit production. The SAST testing ensures that coding standards are compliant with desired standards and injected vulnerabilities are not permitted from coding mishaps by the development team. Automated testing and test reporting shows what is functionally working. With SCA and SAST in place, TAP can generate a comprehensive release report. This report includes:

• Software Bill of Materials (SBOM): Lists all open-source components, licenses, and vulnerabilities.

• Quality Reports: Shows compliance with coding standards and identifies any vulnerabilities.

• Automated Test Reports: Demonstrates that the software functions correctly.

Combining these parts, SBOM, Quality Reports, and Automated Test Reports into a release report demonstrates that quality is a core part of the software development lifecycle. While no software is bug free, this process does provide assurance that the released product should work as designed.

Measuring DevSecOps

As TAP build the DevSecOps pipelines John has identified a way to measure the maturity of the environment. He calls it Mean Time unTil Verification (MTTV). This is the time from when a developer checks code into to the Git repo until it is tested in a production like environment providing feedback that functional requirements are correctly implemented. With quarterly releases of software and no automated tests, MTTV will be measured in months. John explains that research has shown that the sooner a developer can know about a bug, issue, or malfunctioning code, the quicker he/she can fix it and the more reliable the software becomes (Kim, 2019). MTTV provides a way to measure the DevSecOps maturity. John is recommending driving this number below 18 hrs so that problems do not exist for more than one day.

Software Architecture

John, the TAP Security Coach, has brought in solid principles for DevSecOps and software development. He has identified one additional item that is required for secure development to run in a production environment. This last item is a secure software architecture. John explains that certain constructs, design patterns, and techniques need to be applied while the team is developing TRASE. While the team already has extensive logging types of systems being used, John provided the following list of items the developers need to do, watch out for, and incorporate into TRASE as they develop the solution.

· All peer-to-peer connections (process to process) need to be secured from day one. John recommends using certificates for secured socket communication.

· Tie any user interfaces (UI) to active directory for role-based access control before releasing the product. John insists that this has to include internal administrative UI items.

· Database connections cannot straddle the corporate and production networks. John explains that in order to be truly compliant to the ideals of NIST, it is best not to create a shared database between any corporate network database and the production database.

· John is adamant about how to connect to SaaS tools and resources. He first is highly questioning of any of these SaaS tools in a production environment, then he insists that connections must be initiated from within TRASE to the SaaS product and must be as secured as possible. John in general, demonstrates that SaaS in production is a risky idea as it puts the TAP production solution at risk from issues that may happen to the SaaS company (Park, 2024).

Conclusion

NIST as a standard has been available for many years. As companies and solutions become more and more complex, the principles outlined in NIST SP-800 become more important for all solutions. Complying with NIST principles is not enough in today’s hack-centric world. To be as secure as possible solutions must take into account the network (NIST SP-800), the software supply chain, and the way software is designed. As we have seen through TAP, software development can be extremely complex. A company should examine the risks and options for each situation and bring security into every part of the thought process.

Thank you for reading Building and Developing a secure production environment. Comments are always welcome. Post your comments, questions, problems and suggestions at No-Agenda.Tech.

References

Betts, D. B., Manjunath Ennaciri, Hassan Saunderson, Chris. (2023). 3 Essential Steps to Enable Security in DevOps . In: Gartner.

CVE.org. (2024). CVE Numbering Authority (CNA) Operational Rules. CVE Program Mission. 4.0. Retrieved from https://www.cve.org/ResourcesSupport/AllResources/CNARules

Kim, G. (2019). The Unicorn Project: A Novel about Developers, Digital Disruption, and Thriving in the Age of Data: IT Revolution Press.

NIST Special Publication 800-53. Revision 5. Security and Privacy Controls for Information Systems and Organizations. (2020). Gaithersburg, MD: National Institute of Standards and Technology, Attn: Computer Security Division, Information Technology Laboratory Retrieved from https://doi.org/10.6028/NIST.SP.800-53r5

Park, B. P. D. (2024). When SaaS works and when it hurts, putting your business at Risk to save a buck. Retrieved from https://no-agenda.tech/when-saas-works-and-when-it-hurts-putting-your-business-at-risk-to-save-a-buck-part-i