working on it ...

Filters

Explore Public Snippets

Sort by

Found 319 snippets matching: tuning

    public by cghersi  12181  322  6  1

    List of Variables to be set in MySQL for performance tuning

    This is the most important variables to be set in MySQL for performance tuning
    Set global thread_cache_size = 4;
    Set global query_cache_size = 1024*1024*1024;
    Set global query_cache_limit=768*1024;
    Set global query_cache_min_res_unit = 2048;
    Set long_query_time = 5;
    Set global wait_timeout = 7800;
    Set global key_buffer_size = 512*1024*1024;
    Set global table_open_cache = 800;
    SET GLOBAL low_priority_updates=1;
    SET GLOBAL concurrent_insert=ALWAYS;
    SET GLOBAL tmp_table_size = 67108864;
    SET GLOBAL max_heap_table_size = 67108864;
    SET GLOBAL table_definition_cache = 1024;

    public by cghersi  3359  3  6  4

    Change the location of MS SQL Server database files

    When you start playing with serious DB, one of the most important parts is to place (if possible) the files in different drives (physical, not only logical partitions!!!). But what if you already have your DB in production with tons of data on the same location? This script helps in solving the issue. It should be executed in three steps: 1) p
    -- 1) Execute this first chunk to put the DB offline
    ALTER DATABASE MyDBName SET SINGLE_USER WITH ROLLBACK IMMEDIATE;
    ALTER DATABASE MyDBName SET OFFLINE;
    ALTER DATABASE MyDBName MODIFY FILE 
    (
       Name = MyDBName_Data,
       Filename = 'D:\DB\MyDBName.mdf'
    );
    
    ALTER DATABASE MyDBName MODIFY FILE 
    (
       Name = MyDBName_Log, 
       Filename = 'D:\DBLog\MyDBName_log.ldf'
    );
    
    ALTER DATABASE MyDBName MODIFY FILE 
    (
       Name = MyDBName_Idx, 
       Filename = 'E:\DBIndex\MyDBName_log.ndf'
    );
    
    -- 2) Manually move the files in the right location
    
    -- 3) Execute this second chunk to put the DB online
    ALTER DATABASE my SET ONLINE;
    
    ALTER DATABASE my SET MULTI_USER;

    public by cghersi  4136  1  7  0

    Testing script for Stored Procedures

    This script helps developers test the performance of a stored procedure or block of code. Although it was written specifically to assist me in comparing stored procedures, it can easily be modified for other testing purposes. The script expects two tables, 'Seq1' and 'Seq2' to exist, and be of the structure as listed in the header comment of the
    /*************************************************************************************************
    ** File: "20090217 - testing script.sql"
    ** Desc: This is a more generalized script to run performance testing on the sequence 
    ** comparison code. It can be modified to test any other spds. This script requires
    ** the existence of two tables, Seq1 and Seq2, which should have the following structure:
    ** 
    ** CREATE TABLE Seq1 (
    **   CodeLineTxt varchar(max), /* stores the original line of code */
    **   CodeLineNum int not null identity(1,1), /* stores the line number */
    **   MatchLineNum int /* stores the matching line of code from spd #2 */
    ** )
    ** 
    ** Return values: report & results
    ** 
    ** Called by: 
    ** 
    ** Parameters:
    ** Input
    ** ----------
    ** none
    **
    ** Output
    ** -----------
    ** none
    **
    ** Auth: Jesse McLain
    ** Email: jesse@jessemclain.com
    ** Web: www.jessemclain.com
    ** Blog: http://jessesql.blogspot.com/2009/02/comparing-spds-part-3-performance.html
    **
    ** Date: 02/16/2008
    **
    ***************************************************************************************************
    ** Change History
    ***************************************************************************************************
    ** Date:    Author:         Description:
    ** -------- --------        -------------------------------------------
    ** 20080216 Jesse McLain    Created script
    **************************************************************************************************/
    
    SET NOCOUNT ON
    
    DECLARE @RunsPerInputSize int         ; SET @RunsPerInputSize = 1      /* #runs of @InputSize to execute */
    DECLARE @InputSizeIncrement int       ; SET @InputSizeIncrement = 50   /* @InputSize to increment btwn outer runs */
    DECLARE @TotalNumberIncrements int    ; SET @TotalNumberIncrements = 1 /* #increments to execute */
    
    DECLARE @StartTime datetime           ; SET @StartTime = GETDATE()
    DECLARE @StopTime datetime            ; SET @StopTime = GETDATE()
    DECLARE @Seq1 varchar(max)            ; SET @Seq1 = 'Test Sequence "'
    DECLARE @Seq2 varchar(max)            ; SET @Seq2 = 'Test Sequence "'
    DECLARE @TestValue varchar(2)         ; SET @TestValue = ''             /* holder to load values into seq tables */
    DECLARE @InputSize int                ; SET @InputSize = 0              /* input size for the current run */
    DECLARE @InputValueIdx int            ; SET @InputValueIdx = 1          /* counter */
    DECLARE @RunIdx int                   ; SET @RunIdx = 1                 /* counter */
    DECLARE @IncrIdx int                  ; SET @IncrIdx = 1                /* counter */
    DECLARE @Seq1Size int
    DECLARE @Seq2Size int
    DECLARE @Seq1Sizea int
    DECLARE @Seq2Sizea int
    DECLARE @PcntMatch decimal(9, 2)
    
    CREATE TABLE #PerformanceResults (
      InputSize int,
      RunStart datetime,
      RunDone datetime,
      PcntMatch decimal(9, 2)
    )
    
    /* the straight-forward approach to testing would be to start at the smallest
    input size, run through as many runs as we need for that, move on to the next
    input size, test that, until we test the max input size. We don't do that here.
    The problem with that approach is that if there's an external process running
    during the testing of an input size, the results for that size might be false.
    The approach here is to test the min size once, then the next largest size, 
    until the max size is tested, then start all over and repeat until we've tested
    each inputsize as many as "@RunsPerInputSize" times. */
    
    SET @RunIdx = 1
    /* outer loop to increment the number of runs per input size */
    WHILE @RunIdx <= @RunsPerInputSize
    BEGIN
      SET @IncrIdx = 1
      /* inner loop to increment each input size */
      WHILE @IncrIdx <= @TotalNumberIncrements
      BEGIN
        PRINT 'Testing size ' + LTRIM(STR(@IncrIdx)) + '/' + LTRIM(STR(@TotalNumberIncrements))
         + ', for run ' + LTRIM(STR(@RunIdx)) + '/' + LTRIM(STR(@RunsPerInputSize))
     
        SET @InputSize = @IncrIdx * @InputSizeIncrement
     
     
        /* insert rows into test table 1 */
        TRUNCATE TABLE Seq1
        SET @InputValueIdx = 1
        WHILE @InputValueIdx <= @InputSize
        BEGIN
          SET @TestValue = CHAR(FLOOR(RAND() * 10) + 65) + CHAR(FLOOR(RAND() * 10) + 65)
          INSERT INTO Seq1 (CodeLineTxt) VALUES (@TestValue)
          SET @Seq1 = @Seq1 + @TestValue
          SET @InputValueIdx = @InputValueIdx + 1
        END
        UPDATE Seq1 SET MatchLineNum = 0
        SET @Seq1 = @Seq1 + '"'
    
    
        /* insert rows into test table 2 */
        TRUNCATE TABLE Seq2
        SET @InputValueIdx = 1
        WHILE @InputValueIdx <= @InputSize
        BEGIN
          SET @TestValue = CHAR(FLOOR(RAND() * 10) + 65) + CHAR(FLOOR(RAND() * 10) + 65)
          INSERT INTO Seq2 (CodeLineTxt) VALUES (@TestValue)
          SET @Seq2 = @Seq2 + @TestValue
          SET @InputValueIdx = @InputValueIdx + 1
        END
        UPDATE Seq2 SET MatchLineNum = 0
        SET @Seq2 = @Seq2 + '"'
    
    
        /* wrap the executing code around timers to test */
        SET @StartTime = GETDATE()
      
        EXEC spd_SequenceCompare
    
        SET @StopTime = GETDATE()
    
    
        /* record results */
        SELECT @Seq1Size = COUNT(*) FROM Seq1
        SELECT @Seq2Size = COUNT(*) FROM Seq2
      
        SELECT @Seq1Sizea = COUNT(*) FROM Seq1 T1 WHERE MatchLineNum <> 0
        SELECT @Seq2Sizea = COUNT(*) FROM Seq2 T1 WHERE MatchLineNum <> 0
      
        SET @PcntMatch = 100.0 * (@Seq1Sizea / (1.0 * @Seq1Size) + @Seq2Sizea / (1.0 * @Seq2Size)) / 2
    
        INSERT INTO #PerformanceResults (InputSize, RunStart, RunDone, PcntMatch)
        VALUES (@InputSize, @StartTime, @StopTime, @PcntMatch)
     
        SET @IncrIdx = @IncrIdx + 1
      END
    
      SET @RunIdx = @RunIdx + 1
    END
    
    SELECT 
      InputSize, 
      NumberOfRuns = COUNT(*),
      AverageRunTime = AVG(CONVERT(decimal(9, 2), CONVERT(varchar(max), DATEDIFF(ss, RunStart, RunDone))
       + '.' + CONVERT(varchar(max), DATEDIFF(ms, RunStart, RunDone)))),
      AveragePercentMatch = AVG(PcntMatch)
    FROM #PerformanceResults
    GROUP BY InputSize
    
    
    DROP TABLE #PerformanceResults
    
    SET NOCOUNT OFF

    public by cghersi  2515  1  6  0

    Retrieve MySQL data, indexes and log size information

    This SQL retrieves the information regarding the actual size of the data, indexes and log files for the given schema
    select table_schema, table_name,
    concat( round( data_length / ( 1024 *1024 ) , 2 ) , 'Mb' ) data_length_mb,
    concat( round( index_length / ( 1024 *1024 ) , 2 ) , 'Mb' ) index_length_mb,
    concat( round( round( data_length + index_length ) / ( 1024 *1024 ) , 2 ) , 'Mb' ) total_size_mb
    from information_schema.tables
    where table_schema like '<your_schema>'
    order by data_length desc;

    external by luftreich  355  0  3  0

    gevent performance tuning

    gevent performance tuning: gistfile1.txt
    $ vi /etc/sysctl.conf
    # 系统所有进程一共可以打开的文件数量, 每个套接字也占用一个文件描述字
    fs.file-max = 1491124
    # 系统同时保持TIME_WAIT套接字的最大数目,http 短链接会产生很多 TIME_WAIT 套接字。
    net.ipv4.tcp_max_tw_buckets = 7000
    # 关闭 tcp 来源跟踪
    net.ipv4.conf.default.accept_source_route = 0
    # 缩短套接字处于 TIME_WAIT 的时间, 60s -> 30s
    net.ipv4.tcp_fin_timeout = 30
    # 启用 TIME_WAIT 复用,使得结束 TIEM_WAIT 状态的套接字的端口可以立刻被其他套接字使用。
    net.ipv4.tcp_tw_reuse = 1
    net.ipv4.tcp_tw_recycle = 1
    # 关闭 tcp timestamp, 和 tw_reuse/tw_recycle 同时使用
    # tw_recycle 一般不建议使用,RFC1323里面,TCP_TW_RECYCLE和TCP的timestamp选项(timestamp系统默认开启)同时生效的时候,在NAT场景下会导致服务器无法响应连接,这个也是可以复现的。
    net.ipv4.tcp_timestamps = 0
    # 打开内核的 SYN Cookie 功能,可以防止部分 DOS 攻击。
    net.ipv4.tcp_syncookies = 1
    # 减小 tcp keepalive 探测次数,可以即时释放长链接
    net.ipv4.tcp_keepalive_probes = 3
    # 缩短 tcp keepalive 探测间隔时间,同上
    net.ipv4.tcp_keepalive_intvl = 15
    # 增大内核 backlog 参数,使得系统能够保持更多的尚未完成 TCP 三次握手的套接字。
    net.ipv4.tcp_max_syn_backlog = 8388608
    # 同上
    net.core.netdev_max_backlog = 8388608
    # 同上
    net.core.somaxconn = 8388608
    # 默认参数
    net.ipv4.tcp_keepalive_time = 7200
    # 关闭对更大的滑动窗口(如长肥管道)支持,节省系统计算资源
    net.ipv4.tcp_window_scaling = 0
    # 关闭内核对误码大约拥塞的环境(如wifi/3g)的TCP优化,有线线路不需要 tcp_sack
    net.ipv4.tcp_sack = 0
    # 增大应用程序可用端口范围。
    net.ipv4.ip_local_port_range = 1024 65000
    # Increase TCP buffer sizes
    net.core.rmem_default = 8388608
    net.core.rmem_max = 16777216
    net.core.wmem_max = 16777216
    net.ipv4.tcp_rmem = 4096 87380 16777216
    net.ipv4.tcp_wmem = 4096 65536 16777216
    net.ipv4.tcp_congestion_control = cubic
    
    
    $vi /etc/security/limits.conf
    # 增大应用程序的最大打开文件数目限制
    *	hard    nofile          65535
    *	soft    nofile          65535
    
    $ vi /etc/nginx/nginx.conf
    # This number should be, at maximum, the number of CPU cores on your system. 
    worker_processes 8;
    
    # Number of file descriptors used for Nginx. This is set in the OS with 'ulimit -n 200000'
    # or using /etc/security/limits.conf
    worker_rlimit_nofile 200000;
     
    # only log critical errors
    error_log /var/log/nginx/error.log crit
    
    # Determines how many clients will be served by each worker process.
    # (Max clients = worker_connections * worker_processes)
    # "Max clients" is also limited by the number of socket connections available on the system (~64k)
    worker_connections 4000;
    
    # essential for linux, optmized to serve many clients with each thread
    use epoll;
    
    # Accept as many connections as possible, after nginx gets notification about a new connection.
    # May flood worker_connections, if that option is set too low.
    multi_accept on;
     
    # Caches information about open FDs, freqently accessed files.
    # Changing this setting, in my environment, brought performance up from 560k req/sec, to 904k req/sec.
    # I recommend using some varient of these options, though not the specific values listed below.
    open_file_cache max=200000 inactive=20s; 
    open_file_cache_valid 30s; 
    open_file_cache_min_uses 2;
    open_file_cache_errors on;
    
    # Buffer log writes to speed up IO, or disable them altogether
    #access_log /var/log/nginx/access.log main buffer=16k;
    access_log off;
     
    # Sendfile copies data between one FD and other from within the kernel. 
    # More efficient than read() + write(), since the requires transferring data to and from the user space.
    sendfile on; 
     
    # Tcp_nopush causes nginx to attempt to send its HTTP response head in one packet, 
    # instead of using partial frames. This is useful for prepending headers before calling sendfile, 
    # or for throughput optimization.
    tcp_nopush on;
    
    # don't buffer data-sends (disable Nagle algorithm). Good for sending frequent small bursts of data in real time.
    tcp_nodelay on; 
    
    # Timeout for keep-alive connections. Server will close connections after this time.
    keepalive_timeout 30;
    
    # Number of requests a client can make over the keep-alive connection. This is set high for testing.
    keepalive_requests 100000;
     
    # allow the server to close the connection after a client stops responding. Frees up socket-associated memory.
    reset_timedout_connection on;
    
    # send the client a "request timed out" if the body is not loaded by this time. Default 60.
    client_body_timeout 10;
    
    # If the client stops reading data, free up the stale client connection after this much time. Default 60.
    send_timeout 2;
    
    # Compression. Reduces the amount of data that needs to be transferred over the network
    gzip on;
    gzip_min_length 10240;
    gzip_proxied expired no-cache no-store private auth;
    gzip_types text/plain text/css text/xml text/javascript application/x-javascript application/xml;
    gzip_disable "MSIE [1-6]\.";
    
    

    external by kometchtech  228  0  3  0

    kvm host kernel tuning

    kvm host kernel tuning: kvm_host_sysctl
    % sudo sysctl -p[~]
    net.ipv6.conf.all.autoconf = 0
    net.ipv6.conf.all.forwarding = 1
    net.ipv6.conf.br0.forwarding = 1
    net.ipv6.conf.default.forwarding = 1
    net.ipv6.conf.all.accept_ra = 0
    net.ipv4.ip_forward = 1
    net.core.rmem_max = 16777216
    net.core.wmem_max = 16777216
    net.core.rmem_default = 1048576
    net.core.wmem_default = 1048576
    net.ipv4.tcp_wmem = 10240 87380 16777216
    net.ipv4.tcp_rmem = 10240 87380 16777216
    net.ipv4.tcp_sack = 1
    net.ipv4.tcp_no_metrics_save = 1
    net.ipv4.tcp_moderate_rcvbuf = 1
    net.core.somaxconn = 4096
    net.core.netdev_max_backlog = 4096
    net.core.netdev_max_backlog = 30000
    net.ipv4.tcp_congestion_control = htcp
    vm.nr_hugepages = 512
    vm.zone_reclaim_mode = 0
    vm.vfs_cache_pressure = 50
    net.ipv4.tcp_mtu_probing = 1
    net.ipv4.tcp_fin_timeout = 60
    net.ipv4.tcp_tw_reuse = 1
    net.ipv4.tcp_tw_recycle = 0
    net.ipv4.tcp_timestamps = 1
    net.ipv4.tcp_window_scaling = 1
    net.ipv4.ip_local_port_range = 18000    65535
    net.ipv4.tcp_mtu_probing = 1
    kernel.sched_min_granularity_ns = 10000000
    kernel.sched_wakeup_granularity_ns = 15000000
    vm.dirty_ratio = 10
    vm.dirty_background_ratio = 5
    vm.swappiness = 10
    
    

    external by relax-more  137  1  2  0

    [java] Java Performance Tuning memo

    [java] Java Performance Tuning memo: gistfile1.txt
    http://java-performance.info/memory-consumption-of-java-data-types-1/
    
    http://java-performance.info/memory-consumption-of-java-data-types-2/
    
    https://code.google.com/p/memory-measurer/wiki/ElementCostInDataStructures
    
    
    

    external by Steve Scheffler  125  0  2  0

    SQL Server Performance Tuning Class - Day 4

    SQL Server Performance Tuning Class - Day 4: gistfile1.md
    # SQL Server Performance Tuning Class - Day 4
    
    ## How to Measure Your Server
      - 3 Key Numbers
        - How busy is your server
        - How hard is it working
        - Ho wmuch data do you have
    
    ### How Busy is Your Server
      - Perfmon `SQLServer:SQL Statistics - Batch Requests/sec`. Trend this on an hourly basis and break it out by Weekday/Weekend
        - Do NOT look at transactions per/sec, since not all statements are transactions
        - 0 - 1,000 easy to handle with commodity hardware. MAY be possible to run this load on a VM
        - 1,0000 - 5,000 be careful, because a table lock or bad query can knock the server over
        - 5,0000 - 25,0000 You should have several full time DBAs performance tuning. Every index matters crticially
        - 25,000 + requires lots of attention, but you need to have always on availability groups.
        - Record is 150,000
    
    ### How hard is it working
      - Hours of wait time per hour
        - brentozar.com/go/getwaits - trend this on an hourly baiss
        - The more the server works the more it waits
        - 0 - Server isn't doing anything
        - 1 hour of waits - still not doing much
        - 1 hour of waits X # of cores - working hard, look at tuning
    
    ### How much data do you have
      - All numbers below assumce 'commodity' hardware which is 2CPU and 250GB + of RAM
      - 1 - 150GB Std Edition
      - 150 - 500 GB Enterprise
      - 500GB + OLTP vs. Analytical?
      - 1TB OLTP data - very challenging
    
    
    ## What If It's Hardware
      - http://cpuid.com can be used to look at exactly what the CPU is doing
      - Get a faster CPU. Pay attention to the number of cores
      - 7k per core with Enterpise. 2k per core with standard
      - PAGEIOLATCH - make storage faster. Add more memory
      - Perfmon `Phsical Disk: Avg Sec/Read`. 20 - 100 ms is good. > 100ms means you have a storage issue
      - Don't ever change storage in an existing server. Buy new hardware and test/tune there
      - WRITELOG
        - Commit is only completed when the transaction log is written to.
        - Perfmon `Physical Disk: Avg Sec/Write`. 3-20ms is good
        - Perfmon `Physical Disk: Reads/Sec, Writes/Sec`
    
    
    ## Reporting in Production
      - You need more RAM than data. If you have 128GB of data then you should have roughly 192GB of RAM
      - If you can fit data in memory, then most of your tuning problems go away
      - By default, SQL Server will allow one query to allocate up to 25% of available memory. 4 users with aweful reports could swamp a server
      - Can use filtered indexes to get best performance for recent history
      - Indexes views are a view with a where clause - has a clustered index
      - `DBCC TRACEON(610)` minimal logging - useful for DW transfers to significantly drop logging
        - Almost impossible to run if you are running FULL recovery. Need to be running in SIMPLE recovery mode
      - Use SIMPLE recovery model for datawarehouse databases. Pair this with `DBCC TRACEON(610)` to get super minimal logging and better speed for writes
      - `WITH INDEX(1)` will force the clustered index to be used
    
    
    ## How to Tell When tempdb is a Problem
    
      - The 'Version Store' uses tempdb
      - Temp tables and temp variables live here as well
      - SQL server can round robin and distribute amongst the available temp db files
      - For best results, have equally sized files
      - For lots of small allocations, like you have with tempdb, having multiple files helps because you get multiple PFS (page free space) and SGAM (shared global allocation map)
      - PAGELATCH_UP - tempdb waits. means the page is in memory
      - PAGE_IO* - means that page is pulling from disk
      - You can see this with the `sp_WhoIsActive` script
      - Look at wait stats to see if adding tempdb files can help distribute workload
      - SQL 2014 has greatly reduced IO in tempdb
      - Latency thresholds for concern
        - Read latency - 30ms for data files, 5ms for log files
        - Write latency - 30ms for data files, 2ms for log files
      - Microsoft's guidelines - start with the # of tempdb files as you have physical cores, up to 8. From there you need to measure waits and test
      - They like to pre-grow out tempdb and fill up the drive
      - Simple approach - start with 4 equally sized tempdb data files and then watch
      - Don't need to necessarily put tempdb on a separate physical drive unless you're seeing phpyiscal IO waits. Kendra recommended creating a separate logical volume to contain tempdb so that you can control it's growth. Also allows you to move it around easily
      - PREEMPTIVE_OS_* means that it's a wait on something outside of SQL Server. Could also be related to encryption.
    
    
    ## Scale Out for Success
    
      - Content Addressable Storage as a replacement for storing images/blobs in a SQL Server. http://www.emc.com/data-protection/centera.htm
      - ElasticSearch - full text search solution in use by Stack Overflow
      - Redis for an open source caching layer
    
    ## AlwaysOn Availability Groups: Solution, Not Problem
    
      - Not for us
    
    ## Identifying Configuration Problems That Burn Performance
    
      - ostress.exe - simple tool for scripting queries to hit a dev SQL Server. Similar to hammerdb
      - Set autoshrink to off in sys.databases
      - Stats updates happen synchronously when the next read comes through after writes cross the registered threshold
      - THREADPOOL waits are an example of a poison wait. Shouldn't ever really have these.
      - SQL Server memory ools:
        - Execution Plan Cache
        - Buffer Pool Cache (aka Data Cache)
        - Query Workspace Memory  (aka Memory Grant)
      - RESOURCE_SEMAPHORE - Queries are asking for workspace memory and SQL server is unable to grant
        - Scanning large tables can lead to big memory grants
        - For 2012/2014 you need to set memory higher than max memory of 128 on Standard.
        - Make sure that SQL Server is under external memory pressure
      - sp_Blitz checks for the poison wait types
      - The memory limits for SQL Server apply to the Data Cache only. So, add 15% of so to the max edition limit and set the max sql server memory to this level.
      - Leave 10% of 4GB (whichever is higher) for the OS
    
    
    

    external by Mikhail Matyunin  114  0  2  0

    Linux network performance tuning

    Linux network performance tuning: linux_tune.md
    ###Tuning
    
    ```bash
    $ netstat -nao | grep 8080 | grep TIME_WAIT | wc -l
    ```
    
    To achieve the top most performance you should tune the source server system limits:
    
    ```bash
    net.ipv4.tcp_max_tw_buckets = 65536
    net.ipv4.tcp_tw_recycle = 1
    net.ipv4.tcp_tw_reuse = 0
    net.ipv4.tcp_max_syn_backlog = 131072
    net.ipv4.tcp_syn_retries = 3
    net.ipv4.tcp_synack_retries = 3
    net.ipv4.tcp_retries1 = 3
    net.ipv4.tcp_retries2 = 8
    net.ipv4.tcp_rmem = 16384 174760 349520
    net.ipv4.tcp_wmem = 16384 131072 262144
    net.ipv4.tcp_mem = 262144 524288 1048576
    net.ipv4.tcp_max_orphans = 65536
    net.ipv4.tcp_fin_timeout = 10
    net.ipv4.tcp_low_latency = 1
    net.ipv4.tcp_syncookies = 0
    ```
    
    Solution on Linux is:
    
    ```bash
    echo 1 > /proc/sys/net/ipv4/tcp_tw_reuse
    echo 1 > /proc/sys/net/ipv4/tcp_tw_recycle
    ```
    
    This allow the OS to quickly reuse those TIME_WAIT TCP sockets.
    
    Solution on MacOSX is:
    
    ```bash
    sysctl -w net.inet.tcp.msl=1000
    ```
    
    Please note we never tested on Mac, but some people suggest this setting.
    
    

    external by Benjamin Chan  103  0  2  0

    Tuning parallel processing of GLM model fitting

    Tuning parallel processing of GLM model fitting: parallelGLMTuning.md
    # Parallel GLM tuning
    Benjamin Chan  
    Friday, October 23, 2015  
    
    Clear the workspace environment and load packages.
    
    
    ```r
    rm(list=ls())
    gc()
    ```
    
    ```
    ##          used (Mb) gc trigger (Mb) max used (Mb)
    ## Ncells 291977 15.6     407500 21.8   350000 18.7
    ## Vcells 508529  3.9     905753  7.0   785848  6.0
    ```
    
    ```r
    if (!require(devtools)) {install.packages("devtools")}
    ```
    
    ```
    ## Loading required package: devtools
    ```
    
    ```
    ## Warning: package 'devtools' was built under R version 3.1.3
    ```
    
    ```r
    library(devtools)
    source_gist("https://gist.github.com/benjamin-chan/3b59313e8347fffea425")
    ```
    
    ```
    ## Sourcing https://gist.githubusercontent.com/benjamin-chan/3b59313e8347fffea425/raw/c03cd15480a6444399ff5f34892f5911d6e12b44/loadPkg.R
    ## SHA-1 hash of file is 0bb1bb041c1dda15ee613db701d0c5784d1196a5
    ```
    
    ```r
    loadPkg("doParallel")
    ```
    
    ```
    ## Loading required package: doParallel
    ## Loading required package: foreach
    ## Loading required package: iterators
    ## Loading required package: parallel
    ```
    
    ```r
    loadPkg("data.table")
    ```
    
    ```
    ## Loading required package: data.table
    ```
    
    ```
    ## Warning: package 'data.table' was built under R version 3.1.3
    ```
    
    Set up the test data.
    
    
    ```r
    J <- 100  # This is the number of models to fit
    N <- 1E5  # This is the size of the dataset
    i <- rep(1:N, each=J)
    D <- data.table(i,  # id
                    j = rep(1:J, N),  # index repitition
                    x1 = rep(rbinom(N, 1, 0.5), each=J),  # group membership, adult/child
                    x2 = rnorm(N * J),  # fake risk factor
                    x3 = rnorm(N * J),  # fake risk factor
                    x4 = rbinom(N * J, 1, 0.5),  # fake risk factor
                    x5 = rbinom(N * J, 1, 0.5))  # fake risk factor
    D <- D[, logitp := -5 + x1 + x2 + x3 + x4 + x5]
    D <- D[, p := exp(logitp) / (1 + exp(logitp))]
    D <- D[, y := rbinom(N * J, 1, p)]
    D <- rbind(D[, k := j], 
               D[x1 == 0, ][, k := as.integer(J * (x1 + 1) + j)], 
               D[x1 == 1, ][, k := as.integer(J * (x1 + 1) + j)])
    setkey(D, k, i, j)
    rm(i)
    ```
    
    Define the wrapper functions.
    
    
    ```r
    model <- function (D) {
      require(data.table)
      fx <- formula(y ~ x1 + x2 + x3 + x4 + x5)
      M <- glm(fx, data=D, family=binomial)
      fitted(M)
    }
    summarize <- function (t, p) {
      data.frame(workers=workers,
                 user = t[1],
                 sys = t[2],
                 elapsed = t[3],
                 meanPhat = mean(unlist(p)),
                 row.names = NULL)
    }
    ```
    
    Run with various number of workers.
    
    
    ```r
    workers <- 8
    cl <- makeCluster(workers)
    registerDoParallel(cl, cores=workers)
    ptime <- system.time(
      phat <- foreach(K = 1:(J * 3)) %dopar% {
        model(D[k == K, ])
      }
    )
    stopCluster(cl)
    n8 <- summarize(ptime, phat)
    rm(ptime, phat)
    gc()
    ```
    
    ```
    ##             used   (Mb) gc trigger   (Mb)  max used   (Mb)
    ## Ncells    546808   29.3     899071   48.1    818163   43.7
    ## Vcells 150816087 1150.7  326502990 2491.1 320952022 2448.7
    ```
    
    
    ```r
    workers <- 6
    cl <- makeCluster(workers)
    registerDoParallel(cl, cores=workers)
    ptime <- system.time(
      phat <- foreach(K = 1:(J * 3)) %dopar% {
        model(D[k == K, ])
      }
    )
    stopCluster(cl)
    n6 <- summarize(ptime, phat)
    rm(ptime, phat)
    gc()
    ```
    
    ```
    ##             used   (Mb) gc trigger   (Mb)  max used   (Mb)
    ## Ncells    546738   29.2     899071   48.1    899071   48.1
    ## Vcells 150816116 1150.7  326502990 2491.1 320952022 2448.7
    ```
    
    
    ```r
    workers <- 5
    cl <- makeCluster(workers)
    registerDoParallel(cl, cores=workers)
    ptime <- system.time(
      phat <- foreach(K = 1:(J * 3)) %dopar% {
        model(D[k == K, ])
      }
    )
    stopCluster(cl)
    n5 <- summarize(ptime, phat)
    rm(ptime, phat)
    gc()
    ```
    
    ```
    ##             used   (Mb) gc trigger   (Mb)  max used   (Mb)
    ## Ncells    546747   29.2     899071   48.1    899071   48.1
    ## Vcells 150816187 1150.7  326502990 2491.1 320952022 2448.7
    ```
    
    
    ```r
    workers <- 4
    cl <- makeCluster(workers)
    registerDoParallel(cl, cores=workers)
    ptime <- system.time(
      phat <- foreach(K = 1:(J * 3)) %dopar% {
        model(D[k == K, ])
      }
    )
    stopCluster(cl)
    n4 <- summarize(ptime, phat)
    rm(ptime, phat)
    gc()
    ```
    
    ```
    ##             used   (Mb) gc trigger   (Mb)  max used   (Mb)
    ## Ncells    546750   29.2     899071   48.1    899071   48.1
    ## Vcells 150816244 1150.7  326502990 2491.1 320952022 2448.7
    ```
    
    
    ```r
    workers <- 3
    cl <- makeCluster(workers)
    registerDoParallel(cl, cores=workers)
    ptime <- system.time(
      phat <- foreach(K = 1:(J * 3)) %dopar% {
        model(D[k == K, ])
      }
    )
    stopCluster(cl)
    n3 <- summarize(ptime, phat)
    rm(ptime, phat)
    gc()
    ```
    
    ```
    ##             used   (Mb) gc trigger   (Mb)  max used   (Mb)
    ## Ncells    546753   29.2     899071   48.1    899071   48.1
    ## Vcells 150816305 1150.7  326502990 2491.1 320952022 2448.7
    ```
    
    
    ```r
    workers <- 2
    cl <- makeCluster(workers)
    registerDoParallel(cl, cores=workers)
    ptime <- system.time(
      phat <- foreach(K = 1:(J * 3)) %dopar% {
        model(D[k == K, ])
      }
    )
    stopCluster(cl)
    n2 <- summarize(ptime, phat)
    rm(ptime, phat)
    gc()
    ```
    
    ```
    ##             used   (Mb) gc trigger   (Mb)  max used   (Mb)
    ## Ncells    546753   29.2     899071   48.1    899071   48.1
    ## Vcells 150816363 1150.7  326502990 2491.1 320952022 2448.7
    ```
    
    
    ```r
    workers <- 1
    cl <- makeCluster(workers)
    registerDoParallel(cl, cores=workers)
    ptime <- system.time(
      phat <- foreach(K = 1:(J * 3)) %dopar% {
        model(D[k == K, ])
      }
    )
    stopCluster(cl)
    n1 <- summarize(ptime, phat)
    rm(ptime, phat)
    gc()
    ```
    
    ```
    ##             used   (Mb) gc trigger   (Mb)  max used   (Mb)
    ## Ncells    546753   29.2     899071   48.1    899071   48.1
    ## Vcells 150816422 1150.7  326502990 2491.1 320952022 2448.7
    ```
    
    Summarize.
    
    
    ```r
    data.table(models=J,
               size=N,
               rbind(n8, n6, n5, n4, n3, n2, n1))
    ```
    
    ```
    ##    models  size workers  user    sys elapsed  meanPhat
    ## 1:    100 1e+05       8 29.81 174.40  260.29 0.0747293
    ## 2:    100 1e+05       6 23.73 131.30  222.97 0.0747293
    ## 3:    100 1e+05       5 22.71 107.69  189.81 0.0747293
    ## 4:    100 1e+05       4 20.19  86.94  192.11 0.0747293
    ## 5:    100 1e+05       3 16.15  66.54  198.75 0.0747293
    ## 6:    100 1e+05       2  9.91  41.57  223.25 0.0747293
    ## 7:    100 1e+05       1  8.69  17.63  361.23 0.0747293
    ```
    
    
    
    • Public Snippets
    • Channels Snippets