APL timing tests 5.2 to 6.0

I recently began using apl version 6.0.1 and was delighted to discover a major improvement in the execution speed of my major application when comparing it with apl version 5.2.11. After doing many timing tests, it appears that the major improvement is in tieing/untieing files which my app was doing "on the fly". Inspired by this major improvement in execution speed, and also deciding that my file system is sufficiently stable, I decided to change my use of files so that I reference tie numbers, and through quad-mf testing I discovered other ways to improve speed as well. The result is that testing lots of datasets that used to require 122 minutes now requires only 25 minutes, an overall reduction of 80% in execution time. Here are my conclusions about how this improvement was made possible.

My first happy discovery was an improvement from 122 minutes to 79 minutes just by moving from 5.2 to 6.0. That's a 35% reduction in execution time.

I learned that in the 122 minute test that 55% of cpu usage was in Davin's cfDir & cfIO functions which I was using with explicit file names rather than tie numbers.

initial speedup from moving from apl 5.211 to apl 6.01

cpu	aplv	ws	seconds	minutes	pages	/page	datetime	%cf	cf	algos
PIV-2400	5211	QKW_HTM20060827	7320	122	4891	150	20060901	0	0	0

PIV-2400	6010	QKW_HTM20060827	4740	79	4891	97	20060901	0	0	0
PIV-2400	6010	QKW_HTM20060827	4813	80.22	4891	98	20060907.13	0.5108	2458	2355
amd64nm	6010	QKW_HTM20060827	5040	84	4891	103	20060902.22	0.5551	2798	2242
amd64nm	6010	QKW_HTM20060827	2105	35.08	4890	43	20060904.15	0.3196	673	1432

move from PIV-2400 chip to AMD 64 bit chip

cpu	aplv	ws	seconds	minutes	pages	/page	datetime	%cf	cf	algos
amd64nm	5211	QKW_HTM	5520	92	4891	113	20060902.13	0	0	0
amd64nm	5211	QKW_HTM	1710	28.5	4891	35	20060903	0.3443	589	1121
amd64nm	5211	QKW_HTM	1846	30.77	5125	36	20060919.18	0	0	0
amd64nm	5211	QKW_HTM	1890	31.5	5098	37	20060920.1	0.3091	584	1306
amd64nm	5211	QKW_HTM	2520	42	5098	49	20060920.2	0.4076	1027	1493
amd64nm	5211	QKW_HTM	1987	33.12	5098	39	20060920.21	0	0	0
amd64nm	5211	QKW_HTM	2267	37.78	5098	44	20060921.07	0	0	0
amd64nm	5211	QKW_HTM	1760	29.33	5098	35	20060921.07	0	0	0
amd64nm	5211	QKW_HTM	1536	25.6	5098	30	20060921.08	0.3847	591	945
amd64nm	5211	QKW_HTM	1617	26.95	5098	32	20060921.13	0	0	0

PIV-2400	6010	QKW_HTM	2460	41	4891	50	20060901	0	0	0
PIV-2400	6010	QKW_HTM	2532	42.2	4891	52	20060904.05	0.1986	503	2029
PIV-2400	6010	QKW_HTM	2553	42.55	4891	52	20060904.13	0.2039	521	2032
PIV-2400	6010	QKW_HTM	2837	47.28	4891	58	20060904.15	0.2723	773	2064
PIV-2400	6010	QKW_HTM	2831	47.18	4899	58	20060907.16	0	0	0
PIV-2400	6010	QKW_HTM	2794	46.57	5124	55	20060912.1	0	0	0
PIV-2400	6010	QKW_HTM	2695	44.92	5124	53	20060912.14	0.4492	1211	1484
PIV-2400	6010	QKW_HTM	2633	43.88	5124	51	20060912.19	0	0	0
PIV-2400	6010	QKW_HTM	2621	43.68	5125	51	20060915.16	0	0	0
PIV-2400	6010	QKW_HTM	2372	39.53	5098	47	20060921.1	0.1863	442	1930
amd64nm	6010	QKW_HTM	5520	92	4891	113	20060902.19	0	0	0
amd64nm	6010	QKW_HTM	1752	29.2	4891	36	20060903	0.3375	591	1161
amd64nm	6010	QKW_HTM	1740	29	4891	36	20060904.13	0.3481	606	1134
amd64nm	6010	QKW_HTM	1899	31.65	5124	37	20060912.17	0	0	0
amd64nm	6010	QKW_HTM	1847	30.78	5124	36	20060912.17	0.3078	569	1278
amd64nm	6010	QKW_HTM	1793	29.88	5124	35	20060912.18	0	0	0
amd64nm	6010	QKW_HTM	1782	29.7	5125	35	20060915.15	0	0	0
amd64nm	6010	QKW_HTM	1824	30.4	5125	36	20060917.16	0	0	0
amd64nm	6010	QKW_HTM	1561	26.02	5098	31	20060921.09	0.3699	577	984

eliminate conversion of files from .sf to .cf

cpu	aplv	ws	seconds	minutes	pages	/page	datetime	%cf	cf	algos
amd64nm	5211	PVMM	1860	31	4891	38	20060901	0	0	0
amd64nm	5211	PVMM	1670	27.83	4891	34	20060904.11	0.3541	591	1079
amd64nm	5211	PVMM	1537	25.62	5098	30	20060921.19	0.3694	568	969
amd64nm	5211	PVMM	1476	24.6	5098	29	20060923.18	0	0	0
amd64nm	5211	PVMM	1517	25.28	5098	30	20060924.07	0.3801	577	940
amd64nm	5211	PVMM	1492	24.87	5102	29	20061002.19	0	0	0
amd64nm	5211	PVMM	1655	27.58	5106	32	20061011.09	0	0	0
amd64nm	5211	PVMM	1676	27.93	5106	33	20061011.12	0	0	0
amd64nm	5211	PVMM	2195	36.58	5106	43	20061012.21	0	0	0
amd64nm	5211	PVMM	2283	38.05	5106	45	20061013.15	0	0	0
amd64nm	5211	PVMM	2158	35.97	5106	42	20061013.18	0	0	0
amd64nm	5211	PVMM	2175	36.25	5106	43	20061013.19	0.4553	990	1185
amd64nm	5211	PVMM	2261	37.68	5114	44	20061021.16	0	0	0
amd64nm	5211	PVMM	1835	30.58	5101	36	20061214.15	0.344	631	1204

PIV-2400	6010	PVMM	2280	38	4891	47	20060901	0	0	0
PIV-2400	6010	PVMM	2628	43.8	4891	54	20060904.06	0.2277	598	2030
PIV-2400	6010	PVMM	2600	43.33	4891	53	20060904.12	0.2244	583	2017
PIV-2400	6010	PVMM	2565	42.75	4899	52	20060907.18	0	0	0
PIV-2400	6010	PVMM	1986	33.1	5098	39	20060924.07	0.2083	414	1572
amd64nm	6010	PVMM	1680	28	4891	34	20060901	0	0	0
amd64nm	6010	PVMM	1740	29	4891	36	20060902.15	0	0	0
amd64nm	6010	PVMM	1958	32.63	4891	40	20060904.11	0	0	0
amd64nm	6010	PVMM	1712	28.53	4891	35	20060904.12	0.3461	593	1119
amd64nm	6010	PVMM	1582	26.37	5098	31	20060923.17	0	0	0
amd64nm	6010	PVMM	1599	26.65	5098	31	20060924.06	0.3635	581	1018
amd64nm	6010	PVMM	2198	36.63	5106	43	20061013.16	0	0	0
amd64nm	6010	PVMM	2178	36.3	5106	43	20061013.17	0.4336	944	1234
amd64nm	6010	PVMM	1917	31.95	5102	38	20061214.18	0.3262	625	1292
amd64nm	6010	PVMM	1880	31.33	5102	37	20061215.1	0	0	0

confirm with imgindex.w3

cpu	aplv	ws	seconds	minutes	pages	/page	datetime	%cf	cf	algos
amd64nm	5211	IMGINDEX	2670	44.5	2419	110	20060919.2	0	0	0

PIV-2400	6010	IMGINDEX	7244	120.73	2419	299	20060915.19	0	0	0

Dell =
HP =
5211 =
6010 =
tieing =
tn+conv =
tn only =
seconds =
minutes =
pages =
date =
CF % =
CF sec =
algo sec =

One other thing I noticed that is a change in apl behavior. With version 5.2, if you tie a file using its name in mixed case and then you tie it again with the name in all lower case, you will find that it is tied twice with two different tie numbers. With version 6.01, you'll find that it untied the first tie when it did the second tie, so now the first tie number is invalid. This was discovered using Colossal Files.