• Does Element Sequence Matter in XML? It Depends.

    I was reviewing a pull request which integrates an existing app with an external SOAP payment gateway. the gateway looks very simple, given a request like:

    <collectMoney>
        <fromAccount>111</fromAccount>
        <amount>999<amount>
    </collectMoney>
    <more-details>...</more-details>
    

    the payment gateway returns:

    <result>
        <status>success</status>
    </result>
    

    it works fine when I test it with my HTTP client.

    by the WSDL shared by the payment gateway, fromAccount and amount must keep in sequence. however, I found an template inside the pull request: (sequence is different)

    <collectMoney>
        <amount><amount>
        <fromAccount></fromAccount>
    </collectMoney>
    <more-details>...</more-details>
    

    this means the app actually is generating an invalid XML request? How does this work?

    it’s good to see that an app just works, but why??

    I digged into the source code, and this is how it works: create CollectMoney(the class generated by XSD) -> render a XML using the template -> JAXB unmashaller -> CollectMoney instance -> Send to payment gateway via Apache CXF

    I enabled apache CXF loggers to print the outbound/inbound messages, the log shows that the app managed to send out correct sequence! what kind of magic is this?

    it’s the JAXB unmarshaller:

    // copied from JAXB Unmarshaller.java
     * Since unmarshalling invalid XML content is defined in JAXB 2.0,
     * the Unmarshaller default validation event handler was made more lenient
     * than in JAXB 1.0.  When schema-derived code generated
     * by JAXB 1.0 binding compiler is registered with {@link JAXBContext},
     * the default unmarshal validation handler is
     * {@link javax.xml.bind.helpers.DefaultValidationEventHandler} and it
     * terminates the marshal  operation after encountering either a fatal error or an error.
     * For a JAXB 2.0 client application, there is no explicitly defined default
     * validation handler and the default event handling only
     * terminates the unmarshal operation after encountering a fatal error.
    

    what is an error by the definition of W3C?

    //copied from org.xml.ErrorHandler.java
    This corresponds to the definition of "error" in section 1.2
     * of the W3C XML 1.0 Recommendation.  For example, a validating
     * parser would use this callback to report the violation of a
     * validity constraint.  The default behaviour is to take no
     * action.
    

    https://www.w3.org/TR/xml/#sec-terminology show the error:

    [Definition: A violation of the rules of this specification; results are undefined. Unless otherwise specified, failure to observe a prescription of this specification indicated by one of the keywords must, required, must not, shall and shall not is an error. Conforming software may detect and report an error and may recover from it.]

    and it also mentions that a seq:

    content particles occurring in a sequence list must each appear in the element content in the order given in the list.

    so incorrect sequence is an error and by default JAXB 2.0 will continue to process it.

    in my app’s case, an invalid XML request get unmarshalled to java instance, and then the instance gets ‘marshalled’ to an valid XML and then send to the payment gateway.

    Conclusion

    by definition, sequence in a ‘sequence’ does matter. however in the context of unmarshalling, it depends on how does an XML document get unmarshalled.

    Abstraction leaks, the unmarshalling layer doesn’t add much value, but brings unnecessary confusion by leaking incorrect info, the app could directly use the classes generated by the WSDL/XSD.

  • How Does Poor Application Change My Career Path in the Last Few Years

    I have been working as a developer since 2007, most of my time is spent on certain products. however in the last 3 years, I spent most of my time on QA automation and DevOps. this post is the reason behind my switch.

    it was 2015, I was working at an American multinational investment bank, as an application developer. one of the applications is a file mover which moves file from one location to another, such as pulling files from one folder in ServerA and put them to another folder in ServerB.

    the file mover has been heavily used in few markets for few years, it’s a internal built app, so we have all the source codes with very limited unit tests.

    my task was to add support to SFTP so that the file mover can pull files from remote severs via SFTP.

    it looks very simple and well-defined, so I just added my code (and tried not to touch anything else) with unit tests and managed to get approved to production.

    Production issue happened after my release, I was told that “it works fine, but not anymore”, one file on the other side of the earth is not getting moved! after reviewing all the logs and I confirm that it was not caused by my change, one destination was running out of space!

    What I learned

    • nobody knows all features without end-to-end tests. (I should know this from day 1, but I believe that by reading the code I could totally understand the app)A
    • no integration test / end-to-end test, which means the feedback is mostly triggered by production issue.
    • nobody wants to touch the code, not only because the risk of introducing bugs but also taking responsibility for the change

    I refused to make any more changes – until I automated all end-to-end test cases.

    What I did

    • built a BDD testing framework to enable me write all my known test cases in BDD/Gherkin style
    • setup CI pipeline to execute automation tests on each code change
    • documented most of the features in BDD/Gherkin style
    • share the testing freamwork to users, let them to add test cases
    • share BDD with all developers, BDD feature file is mandatory for feature changes

    What I achieved

    after few months:

    • the file mover has a CI pipeline with fully automated end-to-end tests.
    • feature files are considered as documents.
    • developers are comfortable to change the production code.
  • Relaxed Binding of Type-safe Configuration Properties in Spring Boot 1.5

    Question: what’s the output?

    Given this application.yml:

    freeMarker: # with upper-case M
      templatePath: "/freeMarker/upper-case"
     
    freemarker:
      templatePath: "/freemarker/lower-case"
    

    and this configuration bean:

    @Component
    @ConfigurationProperties(prefix = "freeMarker")
    public class FreeMarkerConfigurationSettings {
     
        String templatePath;
     
        public String getTemplatePath() {
            return templatePath;
        }
     
        public void setTemplatePath(String templatePath) {
            this.templatePath = templatePath;
        }
    }
    

    what’s the output of this snippet?

    @Component
    public class PropertiesPrinter implements CommandLineRunner {
     
        @Autowired
        FreeMarkerConfigurationSettings freeMarkerConfigurationSettings;
     
        @Override
        public void run(String... strings) throws Exception {
            System.out.println("templatePath using ConfigurationProperties: " + freeMarkerConfigurationSettings.templatePath);
        }
    }
    

    looks like the property should be resolved to “/freeMarker/upper-case”, but actually it’s not. the output is:

    templatePath using ConfigurationProperties: /freemarker/lower-case

    What? I found a bug in Spring Boot?

    No. actually this is a feature: https://docs.spring.io/spring-boot/docs/current/reference/html/boot-features-external-config.html#boot-features-external-config-relaxed-binding

    Spring Boot uses some relaxed rules for binding Environment properties to @ConfigurationProperties beans, so there does not need to be an exact match between the Environment property name and the bean property name. Common examples where this is useful include dash-separated environment properties (for example, context-path binds to contextPath), and capitalized environment properties (for example, PORT binds to port).

    ok, it’s a feature, then how does Spring boot choose the one? (from a list of candidates?)

    I didn’t find documents with concrete details, so I digged into the source code. Spring did the magic during property binding which is part of the ‘auto-wired’ initialisation: getPropertyValuesForNamePrefix finds the final property value from a list of candidates with a list of “RelaxedNames”, i.e the 2 “propertyValues” we defined in the application.yml:

    private MutablePropertyValues getPropertyValuesForNamePrefix(
          MutablePropertyValues propertyValues) {
       if (!StringUtils.hasText(this.namePrefix) && !this.ignoreNestedProperties) {
          return propertyValues;
       }
       MutablePropertyValues rtn = new MutablePropertyValues();
       for (PropertyValue value : propertyValues.getPropertyValues()) {
          String name = value.getName();
          for (String prefix : new RelaxedNames(stripLastDot(this.namePrefix))) {
             for (String separator : new String[] { ".", "_" }) {
                String candidate = (StringUtils.hasLength(prefix) ? prefix + separator
                      : prefix);
                if (name.startsWith(candidate)) {
                   name = name.substring(candidate.length());
                   if (!(this.ignoreNestedProperties && name.contains("."))) {
                      PropertyOrigin propertyOrigin = OriginCapablePropertyValue
                            .getOrigin(value);
                      rtn.addPropertyValue(new OriginCapablePropertyValue(name,
                            value.getValue(), propertyOrigin));
                   }
                }
             }
          }
       }
       return rtn;
    }
    

    what is the “RelaxedNames” for “freeMarker” we defined in our configuration bean?

    values = {[email protected]} size = 7
     0 = "freeMarker"
     1 = "free_marker"
     2 = "free-marker"
     3 = "freemarker"
     4 = "FREEMARKER"
     5 = "FREE_MARKER"
     6 = "FREE-MARKER"
    

    I would say that “rtn.addPropertyValue()” is not a good name:

    /**
     * Add a PropertyValue object, replacing any existing one for the
     * corresponding property or getting merged with it (if applicable).
     * @param pv PropertyValue object to add
     * @return this in order to allow for adding multiple property values in a chain
     */
    public MutablePropertyValues addPropertyValue(PropertyValue pv) {
       for (int i = 0; i < this.propertyValueList.size(); i++) {
          PropertyValue currentPv = this.propertyValueList.get(i);
          if (currentPv.getName().equals(pv.getName())) {
             pv = mergeIfRequired(pv, currentPv);
             setPropertyValueAt(pv, i);
             return this;
          }
       }
       this.propertyValueList.add(pv);
       return this;
    }
    

    “Add a PropertyValue object, replacing any existing one for the corresponding property or getting merged with it (if applicable).”

    Spring boot resolves the property’s value by comparing each properties with a list of “RelaxedNames” and the return the last one it found.

    Conclusions

    • Relaxed binding is a nice feature, but it could confuse you with the resolved value: spring boot will return the last found property’s value based on the list of “RelaxedNames”
    • Choose better “prefix” names. e.g. “freemarker” is too general, “obsFreemarker” is better.
  • 在新加坡申请泰国旅游签证

    虽然拿中国护照可以在泰国”落地签”, 但是由于家里老人没有单独去其他国家的经历, 加上担心落地签排队, 决定在新加坡的大使馆直接申请一张旅游签证.

    跟其他国家/地区类似, 泰国驻新加坡大使馆也只是负责新加坡居民的签证服务. 譬如EP,PR,LTPV等. 具体信息及流程:https://visaonline.thaiembassy.sg/index.php

    这里单说官方没有提到的细节:

    • 网上填资料申请时可以上传照片
    • 去现场交资料时只需要按照要求打印申请表, 并不需要额外提供照片
    • 去现场交资料时, 家人可以陪着进去
    • 其他人可以代领护照

    时间表:

    • 04-11-2018 23:38:54 visaonline提交资料: 照片(手机拍的) + 护照首页照片 + 机票 + LTPV正反面 + 银行对账单
    • 05-11-2018 傍晚, visaonline状态改为”reviewed”, 提示可以在Nov 6/7去交资料(打印并签名的申请表x2 + 机票等辅助资料复印件x1)
    • 06-11-2018 新加坡假期 放假
    • 07-11-2018 现场提交资料 + 护照, 被告知”collect passort with the receipt tomorrow afternoon 2pm - 3pm” 下午visaonline状态改为”to collect passport”
  • Run Selenium with Headless Chrome in Docker(CentOS)

    As an engineer, I want to test my web app with Chrome in a Jenkins cluster. Chrome is not available in most of Jenkins build nodes, but docker is.

    build image

    based on CentOS, install headless chrome, selenium and chromedriver.

    FROM centos:7
    
    LABEL org.label-schema.schema-version="1.0" \
        org.label-schema.name="Selenium with Headless Chrome and CentOS" \
        org.label-schema.vendor="liguoliang.com" \
        org.label-schema.license="GPLv2" \
        org.label-schema.build-date="20180817"
    
    # install necessary tools
    RUN yum install unzip -y
    RUN curl -O https://bootstrap.pypa.io/get-pip.py
    RUN python get-pip.py
    
    # install headless chrome
    RUN curl -O  https://dl.google.com/linux/direct/google-chrome-stable_current_x86_64.rpm
    RUN yum install google-chrome-stable_current_x86_64.rpm -y
    
    # install selenium
    RUN pip install selenium
    
    # download chromedriver
    RUN mkdir /opt/chrome
    RUN curl -O https://chromedriver.storage.googleapis.com/2.41/chromedriver_linux64.zip
    RUN unzip chromedriver_linux64.zip -d /opt/chrome
    
    # copy the testing python script
    COPY selenium-with-headless-chrome.py .
    RUN python selenium-with-headless-chrome.py
    

    execute selenium from python

    from selenium import webdriver
    from selenium.webdriver.chrome.options import Options
    
    
    chrome_options = Options()
    chrome_options.add_argument("--headless")
    chrome_options.add_argument("--disable-dev-shm-usage")
    chrome_options.add_argument("--no-sandbox")
    
    driver = webdriver.Chrome(r'/opt/chrome/chromedriver', chrome_options=chrome_options)
    
    driver.get("http://python.org")
    print "page title: %s" % driver.title
    

    docker image & source code

    • docker image: https://hub.docker.com/r/guoliangli/selenium-with-headless-chrome-centos/
    • source: https://github.com/guoliang-dev/docker-selenium-with-headless-chrome-centos

    troubleshootings

    Public key for google-chrome-unstable-x.rpm is not installed

    solution: import google key e.g.

    rpm --import https://dl-ssl.google.com/linux/linux_signing_key.pub
    

    for more detials: https://www.google.com/linuxrepositories/

    ERROR Message: unknown error: DevToolsActivePort file doesn’t exist

    add arguments:

     "--headless",
     "--no-sandbox",
     "--disable-dev-shm-usage"
    

    for more details:

    • https://github.com/timgrossmann/InstaPy/issues/2362
    • https://github.com/karma-runner/karma-chrome-launcher/issues/158

subscribe via RSS