Commercial Engineering Software
Structural, hydraulic, geotechnical, electrical, mechanical, transportation, and environmental programs may hide numerical methods, convergence criteria, internal databases, and default assumptions from the user.
Black-Box Engineering: Responsibility to Understand the Design
Engineering Technology and Professional Responsibility
Black-Box Engineering and the Responsibility to Understand the Design
Modern engineers increasingly rely on proprietary software, automated calculations, artificial intelligence, vendor-designed systems, and analytical models whose internal processes may not be fully visible. These tools can improve engineering practice, but they do not transfer professional responsibility away from the engineer.
What Is Black-Box Engineering?
Black-box engineering occurs when an engineer uses a system principally through its inputs and outputs while having limited visibility into—or understanding of—the internal process that produces the result.
The term can apply to calculation software, simulation programs, artificial-intelligence systems, proprietary design tools, vendor selection programs, embedded control systems, automated optimization, digital twins, spreadsheets, and engineering services performed by others.
A black box does not necessarily mean that the system is defective or untrustworthy. Some proprietary tools have undergone extensive verification, validation, certification, and field use. The concern is whether the engineer has enough knowledge and evidence to use the output responsibly for its intended application.
Black-Box Engineering and Black-Box Testing
The general concern about black-box engineering should be distinguished from the established technical method known as black-box testing.
Engineering Concern
Black-Box Reliance
The engineer relies on a result without adequately understanding the system’s assumptions, limitations, governing relationships, validation history, or potential failure modes.
Testing Method
Black-Box Testing
A system is deliberately tested through defined inputs and observed outputs without requiring access to its internal implementation. This can be a legitimate part of verification and validation.
Black-box testing can help establish whether a system performs as required. However, testing only a few expected inputs may not reveal behavior near limits, during abnormal conditions, or outside the domain for which the tool was developed.
NASA identifies black-box, white-box, model-based, functional, stress, and regression testing as distinct software-assurance approaches. A dependable verification program may use several methods rather than relying on a single type of test.
Where Black Boxes Appear in Engineering
Opaque processes can enter a project through many ordinary tools, services, and professional relationships.
Artificial Intelligence
Generative and predictive models may produce convincing calculations, narratives, classifications, or designs without providing a dependable explanation of how the result was developed.
Vendor Selection Tools
Manufacturers may provide proprietary programs for selecting pumps, protective devices, structural components, treatment equipment, HVAC systems, or other products.
Delegated Design
Specialized components or systems may be designed by another engineer. The project engineer may receive only summary calculations, performance criteria, or sealed submittals.
Spreadsheets and Internal Tools
Familiar-looking spreadsheets may conceal incorrect formulas, overwritten cells, undocumented macros, unit errors, obsolete code provisions, and unverified revisions.
Digital Twins and Automated Controls
Real-time models and control systems may combine sensor data, assumptions, algorithms, and automated decisions in ways that are difficult for an operator or reviewing engineer to reconstruct.
Cloud-Based Engineering Services
Online platforms may perform analyses remotely, update algorithms without notice, retain project data, or depend on third-party components not visible to the engineer.
Automated Code Compliance
Software may identify apparent code violations while overlooking exceptions, jurisdictional amendments, unusual occupancy conditions, or requirements that depend on professional interpretation.
Why Black-Box Reliance Is Increasing
Engineering systems have become too complex for every practitioner to independently develop every analytical method, software component, material model, and control algorithm used on a project.
Modern practice necessarily depends on specialization and trusted tools. Engineers routinely use code tables, commercial software, published equations, laboratory data, manufacturer information, and the work of other qualified professionals.
The growth of black-box reliance is being accelerated by cloud computing, subscription software, proprietary algorithms, multidisciplinary project delivery, artificial intelligence, reduced design schedules, and pressure to automate repetitive work.
The Change in the Engineer’s Role
The engineer may spend less time personally performing calculations and more time defining requirements, selecting tools, evaluating assumptions, reviewing outputs, coordinating interfaces, and determining whether the overall result is suitable for its intended use.
This change does not reduce the need for engineering judgment. It moves professional judgment toward oversight, system integration, verification, and risk management.
The Ethical Foundation
Black-box engineering engages several of the most important duties found in professional engineering codes.
Public Protection
Paramount Duty
An engineer should not rely on an opaque process when the resulting uncertainty creates an unreasonable risk to public safety, health, property, or welfare.
Competence
Qualified Use
Competence includes knowing how to use a tool, when it is appropriate, which inputs control the result, and when specialized assistance is needed.
Accountability
Professional Judgment
A software license, vendor warranty, artificial-intelligence disclaimer, or subcontract does not automatically transfer the PE’s professional responsibility.
NSPE’s Board of Ethical Review has considered both computer-assisted design and artificial intelligence. Its opinions recognize technology as an important engineering tool while warning against allowing technology to become a substitute for professional judgment.
The ethical question is not whether the engineer personally wrote the software or derived every equation. It is whether the engineer exercised sufficient competence, direction, review, and control to make a defensible professional decision.
Black Boxes and Responsible Charge
Responsible charge generally requires meaningful professional direction, control, review, and authority over engineering work. It cannot be established solely by reviewing the final output of an unexplained process.
A Sealed Output Is Not Self-Validating
Before signing or sealing engineering documents, a PE should understand the design criteria, governing requirements, material assumptions, boundary conditions, load cases, analytical approach, critical interfaces, and significant limitations.
The engineer need not reproduce every calculation manually. However, the review should be sufficient to identify unreasonable assumptions, inconsistent results, omitted conditions, and departures from applicable standards.
Common Black-Box Failure Modes
An output may appear precise and professionally formatted while still being unsuitable for the intended engineering application.
Incorrect Inputs
The tool may perform its internal process correctly while producing an incorrect result because the engineer entered the wrong units, geometry, material properties, boundary conditions, or design criteria.
Hidden Defaults
Default values may represent a different jurisdiction, code edition, material system, safety factor, environmental condition, or operating assumption.
Use Outside the Valid Domain
A model developed for ordinary conditions may be unreliable for unusual geometry, extreme loading, novel materials, rare events, or conditions outside its calibration data.
False Convergence
Numerical software may terminate normally even though the model is unstable, poorly conditioned, inadequately discretized, or converged to an unintended solution.
Outdated Requirements
Embedded databases and design provisions may not reflect the current code edition, local amendment, adopted standard, or project-specific requirement.
Model or Version Drift
Cloud software and AI systems may change between uses. A result may no longer be reproducible if the model, database, algorithm, or default settings have been updated.
Automation Bias
Engineers may accept a computer-generated result because it appears authoritative, particularly when schedule pressure discourages independent review.
Correlated Verification
A second program may not provide independent verification if it uses the same underlying equations, database, library, vendor engine, or erroneous assumptions.
Interface Failure
Individual components may be correctly designed while their loads, tolerances, controls, utilities, movements, or physical connections are incompatible.
Untraceable Results
The project record may not identify the software version, input file, model configuration, data source, assumptions, or person who reviewed the result.
Artificial Intelligence as an Engineering Black Box
Artificial intelligence intensifies traditional black-box concerns because some systems generate results through models that are difficult to interpret and may not reliably distinguish factual information from plausible invention.
Potential Benefits
Engineering Assistance
- Rapid review of large datasets
- Identification of patterns and anomalies
- Development of preliminary alternatives
- Automation of repetitive documentation
- Assistance with code and literature searches
- Optimization of complex systems
Potential Limitations
Unverified Authority
- Fabricated or inaccurate technical information
- Incomplete explanation of reasoning
- Unknown or outdated source material
- Inconsistent answers to equivalent questions
- Exposure of confidential project information
- Difficulty reproducing a prior result
The NIST Artificial Intelligence Risk Management Framework identifies validity, reliability, safety, security, resilience, accountability, transparency, explainability, interpretability, and privacy as important characteristics of trustworthy AI.
These characteristics provide a useful risk-management framework, but they do not replace engineering standards or professional obligations. The importance of each characteristic depends on the AI system’s intended use and possible consequences.
A Responsible Verification Process
The required level of verification should increase with the uncertainty, novelty, complexity, and possible consequences of error.
1
Define the Intended Use
Determine whether the result will support preliminary planning, final design, safety evaluation, equipment selection, regulatory compliance, or another purpose.
2
Evaluate the Consequences
Identify what could happen if the output is wrong. A drafting convenience does not require the same assurance as a safety-critical design decision.
3
Understand the Technical Basis
Identify the governing relationships, assumptions, code provisions, data sources, limitations, and principal variables affecting the result.
4
Verify the Inputs
Confirm units, geometry, material properties, environmental conditions, load cases, boundary conditions, code editions, and project-specific criteria.
5
Perform Benchmark Checks
Compare representative results with hand calculations, published examples, known solutions, field data, test results, or another suitably independent method.
6
Test Sensitivity and Limits
Vary important inputs and examine whether the output changes in an expected and physically reasonable manner.
7
Review the Output
Examine reactions, equilibrium, trends, warnings, governing conditions, failure modes, and consistency with engineering experience.
8
Document and Preserve
Retain sufficient information to reproduce the analysis, explain the decision, identify the tool version, and support later review.
What Makes Verification Independent?
Repeating a calculation is not necessarily independent verification. The second review must have a reasonable opportunity to detect the type of error that may exist.
Stronger Forms of Verification
- Checking the result with a simplified first-principles calculation
- Using a different analytical method
- Comparing the result with a known benchmark
- Reviewing the model through an engineer who did not create it
- Testing extreme, boundary, and failure conditions
- Comparing model predictions with measured performance
- Performing dimensional, equilibrium, energy, or conservation checks
- Reviewing input data independently from the model operator
If two tools use the same underlying library or assumptions, agreement between them may provide less assurance than expected. Independence should be evaluated technically rather than inferred from different product names.
Vendor Systems and Proprietary Designs
Proprietary design methods are common and can be entirely appropriate. An engineer may not have access to trade secrets or source code, but should obtain enough evidence to evaluate suitability and project integration.
Information That May Be Needed
- Intended applications and excluded uses
- Governing codes, standards, and design criteria
- Required inputs and installation conditions
- Performance limits and environmental restrictions
- Testing, certification, and validation information
- Safety factors and rated capacities
- Inspection and maintenance requirements
- Failure modes and required protective measures
- Interfaces with other project systems
- Identification of the responsible design professional
The project engineer’s responsibility should be clearly distinguished from the delegated engineer’s responsibility. Even where a specialized design is properly delegated, the overall project still requires coordination of interfaces, design criteria, loads, clearances, utilities, movements, and performance requirements.
Traceability and Engineering Records
A defensible engineering result should be traceable to the inputs, assumptions, methods, software version, reviewer, and decision for which it was used.
A Useful Analysis Record May Include
- Tool, program, model, and version used
- Date of analysis and identity of the operator
- Input files and source data
- Units, coordinate systems, and reference datums
- Applicable codes and standards
- Material properties and boundary conditions
- Default settings that were accepted or changed
- Warnings, convergence information, and error messages
- Verification calculations and benchmark results
- Known limitations and unresolved uncertainties
- Reviewer comments and resulting revisions
- Final engineering decision supported by the analysis
Traceability becomes especially important when cloud tools or AI systems may change without preserving the exact model version. Important results should be retained in a form that allows later professional review.
Disclosure and Truthful Communication
Clients, reviewing engineers, and authorities do not ordinarily need a description of every software operation. They should receive information about limitations that could materially affect the interpretation or use of the engineering result.
Material Limitations Should Not Be Hidden
If a model has not been validated for a particular load range, material, geometry, environmental condition, or failure mode, presenting the output as an unqualified final result may be misleading.
Appropriate communication may identify assumptions, uncertainty, excluded conditions, required field verification, operational limitations, or the need for additional analysis.
Disclosure should be proportionate to the significance of the limitation. Minor computational details need not overwhelm the report, but information necessary for safe use and responsible decision-making should not be omitted.
Examples Across Engineering Disciplines
Black-box risk is not confined to artificial intelligence or any single branch of engineering.
Structural Engineering
A three-dimensional analysis produces member forces without clear verification of supports, releases, stiffness assumptions, load paths, second-order effects, or diaphragm behavior. The model’s graphical sophistication does not establish that it represents the actual structure.
Electrical Engineering
A coordination program generates protective-device settings using a proprietary equipment library. The engineer should verify system data, device characteristics, fault assumptions, operating modes, and whether the settings provide appropriate protection and selectivity.
Water Resources Engineering
A hydraulic model predicts acceptable flood elevations, but the result depends on terrain data, roughness, boundary conditions, blocked structures, rainfall assumptions, and calibration. The model output should be evaluated against physical behavior and available observations.
Geotechnical Engineering
A stability program reports an acceptable factor of safety while the selected failure surface, groundwater condition, soil parameters, or analytical method does not represent the critical field condition.
Mechanical Engineering
Manufacturer software selects equipment that satisfies a calculated operating point but does not account for transient loads, system interaction, fouling, unusual fluid properties, installation limitations, or required redundancy.
Transportation Engineering
An optimization system recommends signal timings or roadway changes based on incomplete demand data. The engineer should consider pedestrian needs, emergency operations, unusual traffic patterns, field conditions, and effects not represented in the objective function.
Warning Signs of Excessive Reliance
Certain behaviors suggest that a tool may be functioning as a substitute for professional judgment rather than an aid to it.
Warning Signs Include
- The result is accepted because the software is widely used
- No one can explain the principal assumptions
- The reviewer examines only the final report
- Warnings and convergence messages are routinely ignored
- The result conflicts with physical behavior but is accepted anyway
- Inputs are copied from an earlier project without confirmation
- Verification uses another tool with the same underlying method
- The model is used outside its documented range
- AI-generated references or equations are not independently checked
- Schedule pressure is used to justify omitting review
- No reproducible analysis record is retained
- The person sealing the work cannot defend the result
Frequently Asked Questions
Is using a black-box tool unethical?
Not automatically. Ethical acceptability depends on the intended use, possible consequences, engineer’s competence, available validation, level of review, and treatment of known limitations.
Must an engineer understand the source code?
Usually not. The engineer should understand the technical basis, required inputs, principal assumptions, applicable limits, and evidence supporting the tool’s use for the engineering decision.
Can vendor calculations be accepted?
They may be accepted when prepared under appropriate professional responsibility and supported by sufficient information. The project engineer should still coordinate design criteria and system interfaces.
Can artificial intelligence perform engineering design?
AI can assist with design activities, but its output requires competent professional evaluation. An AI system cannot assume responsible charge or replace the PE’s accountability.
How much checking is enough?
The level of checking should be proportionate to the novelty, uncertainty, complexity, and consequences of error. Safety-critical or unusual applications require stronger evidence than preliminary or low-consequence uses.
Does a software disclaimer remove the engineer’s responsibility?
No. A disclaimer may define the vendor’s obligations, but it does not eliminate the engineer’s duty to use the tool competently and protect the public.
Engineering Technology and Ethics
Continue Exploring Responsible Technology Use
Online-PDH offers continuing-education courses addressing professional judgment, responsible charge, artificial intelligence, automated engineering systems, technical verification, and ethical decisions involving emerging technology.
Last modified: Thursday, 23 July 2026, 9:08 PM