Purpose

My goal in compiling the tips and tricks I've learned into Effective Software Engineering (ESE, pronounced "easy") is to help spread my knowledge, experience, and mistakes, to a broader community that more than likely may not have formal software engineer training. But don’t let my tips and tricks fool you, there are exceptions to every rule. Effective Software Engineering is not meant to be a strict set of rules that guarantee success. Software engineering is just like any other science or engineering discipline in that many decisions require a careful balance of the advantages and disadvantages to one, two, three, or more solutions. The tips and tricks here are general concepts that will improve the project or code you are working on or improve your software development expertise. These suggestions are based purely from my previous experience that I have found beneficial on a non-trivial number ofoccasions.

I want to spread my knowledge, experience, and most importantly my mistakes. A lot of times a well-engineered solution takes time. A lot of time. There is time spent on designing what the code will look like, meaning what classes/object will be created and what will they do and not do. There is time developing interfaces, both externally facing and internally facing interfaces and how they interact with other bits of code. There is time spent on implementing the code, making sure the code does not have "feature creep" or have the potential for feature creep. And lastly, the most time spent in a solid software solution is in the testing. Ideally there should be unit tests, integration tests, and high- level system (or end-to-end tests). Although everyone can agree that comprehensive testing is good, the amount of testing that actually takes place is almost always less than comprehensive. This results in bugs found during operational processing rather than in a controlled development or testing environment.

Most tips and tricks discussed here are meant to be "quick wins", meaning each recommendation should take less than a week to implement and test, and in some cases just a few minutes. These quick wins are things I've compiled over the years as a way to defensively protect the code before bugs are discovered during testing, you might even consider them to be good habits. I've certainly made my own mistakes that have caused bugs in production environments. But by making these mistakes I've matured as a developer who can now spot areas that will cause problems in production environments.

Another reason why I am creating ESE is because a formal software engineering project can be a big and complicated endeavor, but it does not have to be! A formal process that is agreed upon by all developers, managers, customers, and everyone in between can be a really powerful tool. However I want these quick wins to help spread the word that good software doesn't have to be created by someone with formal software engineering training on a team of only software engineers. Anyone can learn to write code, learning to write great bug-free code takes a bit more effort. Formal training helps, but it is not absolutely required. Hence I am giving you my secrets in hopes that you too can learn from my mistakes and build bug-free code!

And lastly, always think about the future of the code rather than a strict process. In my experience, the code will outlive the developer. At any given time, new members join a software project and old members leave. The code stays around making new team members inevitably responsible for the mistakes of their predecessors. It is important for both new and veteran team members to constantly think about what is best for the project. Can I just slap together a little script and push it into the production system? Possibly, but I really shouldn't. The script should be well thought out, various edge cases should be accounted for, and testing should be automatic and repeatable. Without this, bugs will rear their ugly head at the worst possible time.

Feedback!

Effective Software Engineering is and will be a living document. I will periodically add more tips and tricks that may or may not be fully proof read, or fully account for every possible use case. In such a scenario where you find errors, discrepancies, boundary or edge cases where my tips and tricks break down, please let me know. I want to hear from you (Mattermost, RocketChat, email, even Skype), especially if you have something to teach me!

Terminology

The target audience for Effective Software Engineering is mainly for people who have stumbled upon the practice of writing code but do not necessarily have formal software engineering training. Given that audience, there will undoubtedly be unfamiliar terminology that should be understood completely before continuing. This page is set up to enumerate as much terminology as possible that Effective Software Engineering will use. As Effective Software Engineering grows, this page will appear to be a living document as more terms get introduced periodically.

The terminology enumerated on this page contains both software engineering specific terms and terms used to describe common aspects/scenarios of software engineering practices.

Coding

Languages

The terms function, method, procedure, subroutine, and other synonyms, all generally mean the same thing. They are smaller encapsulated bits of logic that may (or may not) accept input variables, perform some computation, then may (or may not) return a value.

In all practicalities, the difference between these definitions are arguable at best. Many languages, particularly object-oriented languages, consider a method different in that it operates on a specific instance object. In Java this is the "this" reference, while in Python this is the "self" reference. Functional programming languages like to use the term "procedure" for indicating the function does not have any side effects. In general I will stick to a convention whereby a "method" implies some sort of object-oriented aspect and the other terms do not. Within the context of Effective Software Engineering, these terms are mostly interchangeable.

Roles

These roles are not mutually exclusive. For example, a developer could also be a sustainer, or a sustainer could also be a tester. These are intended to be more of a logical separation of duties rather than a physical separation of people.

** For the purpose of Effective Software Engineering, the terms "developer", "coder", "programmer", "software developer", and "software engineer" are all synonymous with each other. It is arguable at best that each of these terms have separate definitions, so any references to any of those terms are intended to mean "developer."

Testing

In my experience, very large software projects will have a software development team that writes the code then performs both unit and integration level testing, and a dedicated Test Engineers perform the system testing. It is not unheard of for developers to perform system testing, nor for Test Engineers to dabble in lower level tests, but if a project grows to the point where dedicated Test Engineers are employed then they will more often than not stick with very high level system testing.

References

More Variety

Even though my Effective Software Engineering utilizes Java and Python for example code, many of these tips and tricks focus on the higher level understanding of a concept which could be applied to many other programming languages like C, C++, JavaScript, Ruby, or any other language your project uses. Therefore, here are just a few more resources you can use to develop stronger and more robust software solutions and team practices. These are resources that I stand by, resources that I continue to use periodically.

By Language

TODO Build these references into links!

C

• 21st Century C
• C Programming Essentials
• Intermediate C Programming

C++

• Effective C++
• Effective Modern C++
• Effective STL
• More Effective C++

Java

• Effective Enterprise Java
• Effective Java
• Logback

Javascript

• Effective Javascript

Python

• Effective Python

R

• Effective R Programming

Team Methodologies

• Effective Dev-Ops
• The Scrum Field Guide

Other Software Engineering Topics

• Working Effectively with Legacy Code

Example Code

In these tips and tricks I try to provide most examples with code written in Java and Python demonstrating the topic in a trivial way. There are a select few examples that have sample code written in C/C++, but those cases are for very niche topics – so one may argue the choice of language can play a part in the overall outcome.

I truly believe one of the best ways to learn something new is to take something that works, break it, then figure out a way to make it work again. Because of this philosophy, each example will be complete in that compiling and running the sample code should produce non-erroneous output – assuming you have a proper development environment set up for the language. This does make the example code a bit lengthy, and does add extra fluff like import statements, but I believe it should be easy for you to have a play with the code. Change it, break it, fix it. If nothing else, question it.

To make things even easier, all of the example code is supplied in a few Gitlab projects with README files that detail how to set up the project and run the sample code. These Gitlab links have also been included in the "Space Shortcuts" on the left.

"go easy-cpp" "go easy-java" "go easy-python"

I also try to avoid as much as possible the requirement of third-party libraries. However, for certain tips and tricks it may be required that you have a working development environment to import such a library. With regards to Java, the Gradle build.gradle script defines all extraneous libraries. Running any Gradle build command should pull down the necessary dependencies assuming Gradle was configured with a custom mirror. With regards to Python, I supplied a simple setup.py script that will install the dependencies for you via pip3.

Language Requirements

For the most accurate documentation, each project's README.md file documents the exact development environment required to run the sample code. But generally speaking:

• The C/C++ project depends on CMake version 3.5 or above on your $PATH.

• The Java project requires a JDK version 11 pointed to by your $JAVA_HOME variable. In addition, the Java project uses Gradle as the build and dependency management system. You will need a version of Gradle version 2 or version 4 (but preferably something recent) on your $PATH.

• The Python project requires python3 and pip3 on your $PATH. This project also supplies a setup.py script to make Python dependency management even easier. When you pull down the repository, make sure to run the script to enable any third-party dependencies the same code requires.

Language Choices

In all of the ESE example code projects I have decided to use relatively recent versions of their respective languages. It is very possible to port the examples to older versions, however I would rather focus on the concept each tip/trick is trying to convey rather than the implementation of a particular example. In my opinion, any new code should be written against a recent but stable language version. However there are valid use-cases that absolutely require previous versions, and so I leave it up to the reader to back-port any examples they may find useful in the older standards. I do not perceive that as a challenging exercise, but if you do run into particularly difficult issues I am more than willing to help resolve any back-porting you may endeavor.

C/C++

In many topics the Java and Python example code will suffice. There are however a few niche topics where one may argue the choice in language is important, and getting as "close to bare metal as possible is required." I do not believe that argument holds up in many cases that it is used, so I do have a few topics that dispel that myth with actual C/C++ examples.

Suffice it to say, the C/C++ project will not get very large as it will only have code from very niche topics.

Java

The Java example code provided in ESE was written with a Java 11 development kit. It is possible to tweak the code to support earlier versions of Java, however I decided to stick with this version of Java since it provides developers with more features and utilities to make many examples shorter and more concise. And with Java's fast release cycle, the minimum version of Java required to run the examples will periodically be upgraded.

Also, Java is typically used in Object-Oriented (OO) projects. Since I will make example code short and concise the Java code may not use many OO design principles as that would add extra noise to what I'm trying to get across. In many cases I will choose to convey a particular topic with simple methods and functions than an OO design.

Python 3

The Python example code in ESE was written with a Python 3 interpreter. The main reason why I chose to provide examples in Python 3 was because Python 2 is no longer in active development. Python 2.6 was the last version to genuinely have new features, and Python 2.7 was created as an interim version that back-ported several Python 3 features to Python 2. The idea was that Python 2.7 will help ease the pain when converting a project from Python 2 to 3. I should not even have to say this, but unfortunately there are still many Python 2-only projects running around. Hopefully ESE puts pressure on them to upgrade their compatibility with Python 3.

Lastly, there are multiple implementations of Python that are generally compatible with one another. CPython and Jython are among two industry standard implementations. CPython is the official implementation with the interpreter written in C, while Jython is another popular Python implementation but the interpreter is written in Java. In many cases, it might not matter which implementation you are using, but it is very important to know the strengths and limitations of the interpreter you decide to use because your application or script may genuinely behave differently between the two. The examples and descriptions provided in ESE were tested with CPython.

Upcoming Topics

There are many, many, topics that I have not yet created content for. I will be creating so much more tips and tricks in the future, most of which I already have an idea of what I want to share while other tips and tricks just have not yet popped in my head yet. So please watch this space to receive email notices when I publish new topics!! Or if you're interested in a particular topic, shoot the idea by me and I'd be more than happy to add that to the top of my queue.

• Collaboration
  • Code Reviews
• Configuration Files
  • File Formats
  • Prefer Configuration Files over Code
  • Version Control Configuration Files
• Databases
  • Examples
    • MongoDB
    • SQLite
  • SQL vs. NoSQL
• Development Tools
  • IDE vs. CLI
  • Build Tools for Java
    • Gradle
    • Maven
    • Nexus
  • Build Tools for Python
    • Distutils and Setuptools
• Design
  • "Anti Patterns"
  • "Design Patterns"
    • Decorator Pattern
    • Null Object Pattern
    • Singleton Pattern
  • Encapsulation
  • Global Variables
  • Immutability
  • SOLID
  • Stateless Code
• Embrace the Language
  • Coding Style and Standards
  • Language Supplied Utilities
  • Software Reuse!
• Error Handling
  • Exception Handling
  • Library Exceptions API
  • Signal Handling
  • Using "null"
• Interfaces
  • External Facing Interfaces
  • Internal Facing Interfaces
• Inter-Process Communication
  • Data Formats
  • RabbitMQ
  • Sockets
• Memory Management
  • Memory Leaks
  • Strong vs. Weak References
• Methodologies
  • Scrum
  • Test Driven Development
  • Waterfall
• Myths
  • Python Threads are Useless
• Optimizations
  • Profiling Java
  • Profiling Python
• Programming Paradigm
  • Functional Programming
  • Object Oriented Programming
• Releasing Software
  • Building Deliverables (zip, rpm, deb, Docker images, etc...)
  • Semantic Versions (SemVer)
• Security
  • Hard Coding Cryptographic Keys
  • Protecting Against Database Timing Attacks
  • Storing Passwords
• Testing
  • Continuous Integration
  • Integration Testing
  • Mock Objects
• User Experience
• No Good $HOME
  • Callbacks
  • Serialization
  • Systemd Files
  • Variable Arguments