Java course IAG0040 in Tallinn Technical University

Now is the time to make an official announcement: I am no longer teaching Java to Master's students in Tallinn Technical University (TTÜ).

I have really enjoyed doing it during the past 6 years, and I definitely learned a lot during this time.
Teaching in a university is not something you would do for money and it also eats up a lot of your spare time - I did it because I know I am good at it and I wanted to share my knowledge and experience in the field, especially considering the fact that most university professors/lecturers have little understanding of how software development really works and how to teach it. I did it for those students each year who really had their eyes shining. Because of them, it was worth it.

However, Java is no longer the coolest programming language out there, it slowly dies due to lack of or very slow development. Its features are outdated, it is not as productive as I would like it to be. It's time to move on. Probably Java still is one of the best programming languages to teach in the universities, but students should understand that world software development field is evolving quickly and they need to keep an eye of what is happening in the industry.

I am still a big fan of JVM - it is well tuned for performance and it is cross-platform. .NET/C# is not nearly an option. I really like Scala, Clojure seems very interesting, as well as Vala, Go and Dart. Kotlin seems very promising as well. Sooner or later there will be another popular language on JVM that most Java developers will be able to shift to. Hopefully there won't be too many of such languages and hopefully they will be statically typed. I recommend every Java developer to check the Play framework for a fresh look at Java. Anyway, the current trend in software development is to merge academic computer science and industrial programming together again - you should pay a lot more attention to functional programming in addition to the good old object oriented programming (OOP) now. The concept of DevOps is also becoming more and more important, meaning that soon you will not be able to survive as a developer, if you don't know how to create a software product from scratch until the end, deploying it yourself to the production system.

However, I am not going to leave the teaching 'business'. I will still give talks at conferences, organize trainings, and hopefully contribute to making IT education better with Codeborne Academia, but more on that later. I am still open to offers from universities and colleges as well, feel free to contact me.

Now the important part

All the code written during the course by me and the students during the past 6 years is now available on the Github:
https://github.com/angryziber/java-course

The lastest (2011) lecture slides are available on Slideshare:
http://www.slideshare.net/antonkeks/presentations

Or go to specific lectures using the links below:

1. Introduction
2. Java basics, program flow
3. OOP in Java
4. Exceptions and Collections
5. Generics, Enums, Assertions
6. Unit testing and Agile software development
7. Text processing, Charsets & Encodings
8. I/O, Files, Streams
9. Networking, Reflection
10. Threads and Concurrency
11. Design Patterns
12. Web, Servlets, XML
13. JDBC, Logging
14. Java Beans, Applets, GUI
15. Advacned: Ant, Scripting, Spring & Hibernate

Database Refactoring

A couple of months ago I have made a short keynote titled Dinosaur Strategies: How Can Data Professionals Still Prosper in Modern Organisations, inspired by Scott Ambler's joke on the fictional Waterfall 2006 conference website.

(see the slides)

I primarily deal with 'application' aspects of software development using Agile practices, so I have a hard time understanding how some Data Professionals can be so behind in their evolution, and not doing some basic things like iterative development, unit tests, continuous integration, etc.

Last week I was asked to give a talk on Database Refactoring. The topic seemed challenging enough and as no Database Professionals cared to lead the topic, I decided to give it a try. The result is a motivational speech for both database developers as well as others in the software development process.

I have discussed the cultural conflict of database and OOP developers, the problem of refactoring tools available to relational database developers lagging behind, and some solutions to these problems that can help before these tools become available:

(1) Development Sandboxes
(2) Regression Testing
(3) Automatic Changelog, Delta scripts
(4) Proper Versioning
(5) Continuous integration
(6) Teamwork & Cultural Changes

Other discussed topics include Refactoring of Stored Code vs Database Schema, Agile Reality, Overspecialization (016n), Database not being under control, Database Smells, Fear of Change, Scenarios, Dealing with Coupling, Dealing with unknown applications, Proper versioning, Continuous Integration using sandboxes, and Delta Scripts (Migrations), which make evolutionary database schema possible.

The dinosaurs below are the reminder of my previous keynote available above. They come from the very nice Dinosaurs Song, available on YouTube, which I have actually played after the keynote itself.

Below are full slides of the Database Refactoring talk.

(click for PDF slides)

Excessive memory usage by Oracle driver solved

On my day job I deal with Internet banking. The Internet bank is a relatively large and high-load Java/Spring/Hibernate web application, which uses Oracle databases.

During our recent transition from a centralized data accessor (VJDBC) to local JDBC connection pools to reduce data routrip times, we have started having issues with memory usage in our application servers: some requests started to allocate tens to hundreds of megabytes of memory. While Garbage Collector was successfully reclaiming all this memory afterwards (no memory leaks), it still posed a problem of high peak memory usage as well as too frequent collections, also affecting the overall performance.

While profiling memory allocations with JProfiler, I have discovered that OracleStatement.prepareAccessors() is responsible for these monstrous allocations (up to 600 Mb at once, most in either char or byte giant arrays). Google has pointed to this nice article on reducing the default prefetch size, describing a very similar situation, however these guys have had problems with queries returning LOBs. We haven't used any LOBs in our problematic queries and haven't modified the defaultRowPrefetch connection property knowingly.

Further investigation led to the way we were using Hibernate: for some quesries that are expected to return large result sets, we were using the Query.setFetchSize() or Criteria.setFetchSize() methods with rather high values (eg 5000). This seemed reasonable, because we were also using the setMaxResults() method with the same value to reduce the maximum length of the returned ResultSet. However, after doing some upgrades of Java, Hibernate, and Oracle driver, this had started having these memory allocation side-effects. It seems that now Hibernate translates this fetchSize parameter directly to OracleStatement's rowPrefetch value, forcing it instantly allocate a rowPrefetch*expectedRowSize sized array even before it runs the actual query. This array can be ridicuosly large, even if the actual query returns only a few rows afterwards. Later investigation showed that also having the batch-size attribute in the Hibernate mapping files (hbm.xml) has exactly the same effect and also results in giant pre-allocations.

As a result, we had to review all batch-size and setFetchSize() values that we were using with our Hibernate queries and mappings, in most cases reducing them significantly. This would reduce the worst-case performance of some long queries (they would require more roundtrips to the database), but would also reduce the overall amount of garbage accumulating in the heap and thus reduce the frequency of garbage collections, having a positive impact on CPU load. Shorter results would run equally fast, so it makes sense actually to rely on average statictics of the actual responses when chosing optimal rowPrefetch values. The default value is 10, which is hardcoded in the Oracle driver.

For longer queries, the abovementioned article has proposed an idea of geometrically increasing the rowPrefetch (setting it twice as big for each subsequent fetch manually). This is a nice idea, but I wonder why Oracle driver can't do this automatically? This is how Java collections behave when they resize themselves. I haven't tried doing this with Hibernate yet, but I think it should be possible, especially if you use the Query.scroll() instead of Query.list().