• How Does Poor Application Change My Career Path in the Last Few Years

    I have been working as a developer since 2007, most of my time is spent on certain products. however in the last 3 years, I spent most of my time on QA automation and DevOps. this post is the reason behind my switch.

    it was 2015, I was working at an American multinational investment bank, as an application developer. one of the applications is a file mover which moves file from one location to another, such as pulling files from one folder in ServerA and put them to another folder in ServerB.

    the file mover has been heavily used in few markets for few years, it’s a internal built app, so we have all the source codes with very limited unit tests.

    my task was to add support to SFTP so that the file mover can pull files from remote severs via SFTP.

    it looks very simple and well-defined, so I just added my code (and tried not to touch anything else) with unit tests and managed to get approved to production.

    Production issue happened after my release, I was told that “it works fine, but not anymore”, one file on the other side of the earth is not getting moved! after reviewing all the logs and I confirm that it was not caused by my change, one destination was running out of space!

    What I learned

    • nobody knows all features without end-to-end tests. (I should know this from day 1, but I believe that by reading the code I could totally understand the app)A
    • no integration test / end-to-end test, which means the feedback is mostly triggered by production issue.
    • nobody wants to touch the code, not only because the risk of introducing bugs but also taking responsibility for the change

    I refused to make any more changes – until I automated all end-to-end test cases.

    What I did

    • built a BDD testing framework to enable me write all my known test cases in BDD/Gherkin style
    • setup CI pipeline to execute automation tests on each code change
    • documented most of the features in BDD/Gherkin style
    • share the testing freamwork to users, let them to add test cases
    • share BDD with all developers, BDD feature file is mandatory for feature changes

    What I achieved

    after few months:

    • the file mover has a CI pipeline with fully automated end-to-end tests.
    • feature files are considered as documents.
    • developers are comfortable to change the production code.
  • Relaxed Binding of Type-safe Configuration Properties in Spring Boot 1.5

    Question: what’s the output?

    Given this application.yml:

    freeMarker: # with upper-case M
      templatePath: "/freeMarker/upper-case"
     
    freemarker:
      templatePath: "/freemarker/lower-case"
    

    and this configuration bean:

    @Component
    @ConfigurationProperties(prefix = "freeMarker")
    public class FreeMarkerConfigurationSettings {
     
        String templatePath;
     
        public String getTemplatePath() {
            return templatePath;
        }
     
        public void setTemplatePath(String templatePath) {
            this.templatePath = templatePath;
        }
    }
    

    what’s the output of this snippet?

    @Component
    public class PropertiesPrinter implements CommandLineRunner {
     
        @Autowired
        FreeMarkerConfigurationSettings freeMarkerConfigurationSettings;
     
        @Override
        public void run(String... strings) throws Exception {
            System.out.println("templatePath using ConfigurationProperties: " + freeMarkerConfigurationSettings.templatePath);
        }
    }
    

    looks like the property should be resolved to “/freeMarker/upper-case”, but actually it’s not. the output is:

    templatePath using ConfigurationProperties: /freemarker/lower-case

    What? I found a bug in Spring Boot?

    No. actually this is a feature: https://docs.spring.io/spring-boot/docs/current/reference/html/boot-features-external-config.html#boot-features-external-config-relaxed-binding

    Spring Boot uses some relaxed rules for binding Environment properties to @ConfigurationProperties beans, so there does not need to be an exact match between the Environment property name and the bean property name. Common examples where this is useful include dash-separated environment properties (for example, context-path binds to contextPath), and capitalized environment properties (for example, PORT binds to port).

    ok, it’s a feature, then how does Spring boot choose the one? (from a list of candidates?)

    I didn’t find documents with concrete details, so I digged into the source code. Spring did the magic during property binding which is part of the ‘auto-wired’ initialisation: getPropertyValuesForNamePrefix finds the final property value from a list of candidates with a list of “RelaxedNames”, i.e the 2 “propertyValues” we defined in the application.yml:

    private MutablePropertyValues getPropertyValuesForNamePrefix(
          MutablePropertyValues propertyValues) {
       if (!StringUtils.hasText(this.namePrefix) && !this.ignoreNestedProperties) {
          return propertyValues;
       }
       MutablePropertyValues rtn = new MutablePropertyValues();
       for (PropertyValue value : propertyValues.getPropertyValues()) {
          String name = value.getName();
          for (String prefix : new RelaxedNames(stripLastDot(this.namePrefix))) {
             for (String separator : new String[] { ".", "_" }) {
                String candidate = (StringUtils.hasLength(prefix) ? prefix + separator
                      : prefix);
                if (name.startsWith(candidate)) {
                   name = name.substring(candidate.length());
                   if (!(this.ignoreNestedProperties && name.contains("."))) {
                      PropertyOrigin propertyOrigin = OriginCapablePropertyValue
                            .getOrigin(value);
                      rtn.addPropertyValue(new OriginCapablePropertyValue(name,
                            value.getValue(), propertyOrigin));
                   }
                }
             }
          }
       }
       return rtn;
    }
    

    what is the “RelaxedNames” for “freeMarker” we defined in our configuration bean?

    values = {[email protected]} size = 7
     0 = "freeMarker"
     1 = "free_marker"
     2 = "free-marker"
     3 = "freemarker"
     4 = "FREEMARKER"
     5 = "FREE_MARKER"
     6 = "FREE-MARKER"
    

    I would say that “rtn.addPropertyValue()” is not a good name:

    /**
     * Add a PropertyValue object, replacing any existing one for the
     * corresponding property or getting merged with it (if applicable).
     * @param pv PropertyValue object to add
     * @return this in order to allow for adding multiple property values in a chain
     */
    public MutablePropertyValues addPropertyValue(PropertyValue pv) {
       for (int i = 0; i < this.propertyValueList.size(); i++) {
          PropertyValue currentPv = this.propertyValueList.get(i);
          if (currentPv.getName().equals(pv.getName())) {
             pv = mergeIfRequired(pv, currentPv);
             setPropertyValueAt(pv, i);
             return this;
          }
       }
       this.propertyValueList.add(pv);
       return this;
    }
    

    “Add a PropertyValue object, replacing any existing one for the corresponding property or getting merged with it (if applicable).”

    Spring boot resolves the property’s value by comparing each properties with a list of “RelaxedNames” and the return the last one it found.

    Conclusions

    • Relaxed binding is a nice feature, but it could confuse you with the resolved value: spring boot will return the last found property’s value based on the list of “RelaxedNames”
    • Choose better “prefix” names. e.g. “freemarker” is too general, “obsFreemarker” is better.
  • 在新加坡申请泰国旅游签证

    虽然拿中国护照可以在泰国”落地签”, 但是由于家里老人没有单独去其他国家的经历, 加上担心落地签排队, 决定在新加坡的大使馆直接申请一张旅游签证.

    跟其他国家/地区类似, 泰国驻新加坡大使馆也只是负责新加坡居民的签证服务. 譬如EP,PR,LTPV等. 具体信息及流程:https://visaonline.thaiembassy.sg/index.php

    这里单说官方没有提到的细节:

    • 网上填资料申请时可以上传照片
    • 去现场交资料时只需要按照要求打印申请表, 并不需要额外提供照片
    • 去现场交资料时, 家人可以陪着进去
    • 其他人可以代领护照

    时间表:

    • 04-11-2018 23:38:54 visaonline提交资料: 照片(手机拍的) + 护照首页照片 + 机票 + LTPV正反面 + 银行对账单
    • 05-11-2018 傍晚, visaonline状态改为”reviewed”, 提示可以在Nov 6/7去交资料(打印并签名的申请表x2 + 机票等辅助资料复印件x1)
    • 06-11-2018 新加坡假期 放假
    • 07-11-2018 现场提交资料 + 护照, 被告知”collect passort with the receipt tomorrow afternoon 2pm - 3pm” 下午visaonline状态改为”to collect passport”
  • Run Selenium with Headless Chrome in Docker(CentOS)

    As an engineer, I want to test my web app with Chrome in a Jenkins cluster. Chrome is not available in most of Jenkins build nodes, but docker is.

    build image

    based on CentOS, install headless chrome, selenium and chromedriver.

    FROM centos:7
    
    LABEL org.label-schema.schema-version="1.0" \
        org.label-schema.name="Selenium with Headless Chrome and CentOS" \
        org.label-schema.vendor="liguoliang.com" \
        org.label-schema.license="GPLv2" \
        org.label-schema.build-date="20180817"
    
    # install necessary tools
    RUN yum install unzip -y
    RUN curl -O https://bootstrap.pypa.io/get-pip.py
    RUN python get-pip.py
    
    # install headless chrome
    RUN curl -O  https://dl.google.com/linux/direct/google-chrome-stable_current_x86_64.rpm
    RUN yum install google-chrome-stable_current_x86_64.rpm -y
    
    # install selenium
    RUN pip install selenium
    
    # download chromedriver
    RUN mkdir /opt/chrome
    RUN curl -O https://chromedriver.storage.googleapis.com/2.41/chromedriver_linux64.zip
    RUN unzip chromedriver_linux64.zip -d /opt/chrome
    
    # copy the testing python script
    COPY selenium-with-headless-chrome.py .
    RUN python selenium-with-headless-chrome.py
    

    execute selenium from python

    from selenium import webdriver
    from selenium.webdriver.chrome.options import Options
    
    
    chrome_options = Options()
    chrome_options.add_argument("--headless")
    chrome_options.add_argument("--disable-dev-shm-usage")
    chrome_options.add_argument("--no-sandbox")
    
    driver = webdriver.Chrome(r'/opt/chrome/chromedriver', chrome_options=chrome_options)
    
    driver.get("http://python.org")
    print "page title: %s" % driver.title
    

    docker image & source code

    • docker image: https://hub.docker.com/r/guoliangli/selenium-with-headless-chrome-centos/
    • source: https://github.com/guoliang-dev/docker-selenium-with-headless-chrome-centos

    troubleshootings

    Public key for google-chrome-unstable-x.rpm is not installed

    solution: import google key e.g.

    rpm --import https://dl-ssl.google.com/linux/linux_signing_key.pub
    

    for more detials: https://www.google.com/linuxrepositories/

    ERROR Message: unknown error: DevToolsActivePort file doesn’t exist

    add arguments:

     "--headless",
     "--no-sandbox",
     "--disable-dev-shm-usage"
    

    for more details:

    • https://github.com/timgrossmann/InstaPy/issues/2362
    • https://github.com/karma-runner/karma-chrome-launcher/issues/158
  • 'Software Testing Foundations' Notes

    randomly copied from: Software Testing Foundations, 4th Edition

    The Fundamental Test Process

    Test Planing and Control

    Planning of the test process starts at the beginning of the software development project. The mission and objectives of testing must be defined and agreeed upon as well as the resources necessary for the test process.

    The main task of planning is to determine the test strategy or approach. since an exhaustive test is not possible, priorities must be set based on risk assessment. The test activities must be distributed to the individual subsystems, denpending on the expected risk and the sverity of failure effects.

    Test Analysis and Design

    The first task is to review the test basis, i.e., the specification of what should be tested. the specification should be concrete and clear enough to develop test cases. the basis for the creation of a test can be the specification or architecure documents.

    it is important to ensure traceability between the specifications to be tested and the tests themselves. it must be clear which test cases test which requirement and vice versa. Only this way is it possible to decide which requirements are to be or have been tested, how intensively and with which test cases.Even the traceability of requirement changes to the test cases and vice versa should be verified.

    Test Implementation and Execution

    Tests must be run and logged. the priority of the test cases decided during planning.

    Test Evaluation and Reporting

    During test evaluation and reporting, the test object is assessed against the set test exit criteria specified during planning. this ma result in normal termination of the tests if all criteria are met, or it may be decided that additional test cases should be run or that the criteria weree too hard. it must be decided whether the test exit criteria defined in the test plan are fulfilled.

    Test Closure Activities

    the following data should be recorded:

    • when wwas the software system released?
    • when was the test finished or terminated?
    • when was a milestone reached or a maintenane release completed? Importeant informantion for evaluation can be extracted by asking the following questions:
    • which planned results are achieved and when – if at all?
    • which unexpected events happened (reasons and how they were met)?
    • are there any open problems? and change requests? why ere thye not implemented?
    • how was user acceptance after deploying the system?

    General Principles of Testing

    • Testing shows the presence of defects, not their absence. Testing can show that the product fails, cannot prove that a program is defect free. even if no failures are found during testing, this is no proof that there are no defects.
    • Exhaustive tesing is impossible it’s impossible to run an exhaustive test that includes all possible values.
    • Testing activities should start as early as possible
    • Defect clustering. if many defects are detected in one place, there are normally more defects nearby.
    • The pesticide paradox. new and modified cases should be developed and added to the test.
    • Testing is context dependent.
    • No failures means the system is useful is a fallacy.

    Test Plan

    The test manager might participate in the following planning activities:

    • defininingthe overall approach to and strategy for testing
    • deciding about the test environment and test auotomation
    • defining the test level and their interaction and integrating the testing activities with other project activities
    • deciding how to evaluate the test results
    • selecting matrics for monitoring and controlling test work, as well as defining test exit criteria
    • determining how much test documentation shall be prepared and determining templates
    • writing the test plan and deciding on what, who, when and how much testing
    • estimating test effor and test cost.

    Test Entry and Exit Criteria

    typial entry criteria:

    • the test environment is ready
    • the test tools are ready for use in the test environment
    • test objects are installed in the test environment
    • the necessary test data is available

    exit:

    • achieved test coverage: tests run, covered requirements, code coverage etc.
    • product quality: defect density, defect severity, failure rate and reliability of the test object
    • residual risk: tests not executed, defects not repaired, incomplete coverage of requirements or code. ect.
    • economic constraints: allowed cost, project risks, release deadlines and market chance.

    Test Plan according to IEEE 829-1998

    • Test Plan Identifier
    • Introduction
    • Test Items
    • Features to be Tested
    • Features not to be Tested
    • Approach
    • Item Pass/Fail Criteria (exit criteria)
    • Suspension Criteria and Resumption Requirements
    • Test Deliverables
    • Testing Tasks
    • Environmental Needs
    • Staffing and Training Needs
    • Schedule
    • Risks and Contingencies
    • Approvals

subscribe via RSS