SSIS Package with Dynamic ADO Connection

SSIS Package with Dynamic ADO Connection.

I have created a package that collects database information from different instances all over the network and stored in one central database table.

For this, I have collected Server Name, Database Name, Logical File Name, Physical File Name and File Size in MB.

I have stored all instance info in my central database table.

Table Script.

Create Table SQLServerInstances
(

    ServerName Varchar(20),
    DBName Varchar(20),

    UserName Varchar(20),

    PWD Varchar(20)
)
Insert Into SQLServerInstances Values('yourLocalMachineName','AdventureWorks2012','sa','password')

 I had taken one Execute SQL Task to get list of instances then move it to Foreach loop and store the values to database table using Data Flow Task.

Step1 : Drag Execute SQL Task, Foreach Loop Container and Data Flow task under Foreach Loop Container.

DynamicConnection 1

Step2 : Create Variables.

–> Obj_Result : DataType Object.

–> ServerName : DataType String.

–> DBName : DataType String.

–> UserName1 : DataType String.

–> PWD : DataType String.

DynamicConnection 2

Step3 : Right click on Execute SQL Task –> Edit and fill the details as shown in below figure.

DynamicConnection 3

Step4 : Select Result Set on left pane of Execute SQL Task and select your object variable.

DynamicConnection 4

Step5 : Right Click on Foreach Loop Container –> Edit.

Select Foreach ADO Enumerator in Enumerator and Select Obj_Result in ADO Object source varialbe field.

DynamicConnection 5

Step6 : Select Variable Mapping on left pane of Foreach Loop Container and select varialbes as you see in below figure.

DynamicConnection 6

Step7 : Now Right click on Data Flow Task –> Edit.

DynamicConnection 7

Step8 : Right click on OLE DB source. create one OLEDB data connection with name Dynamic Connection with your local database connection.

SQL Command.


SELECT @@ServerName as ServerName,
   DB_NAME(database_id) AS DatabaseName,
   Name AS LogicalName ,
   Physical_Name as PhysicalName,
   (size*8)/1024 DBSize
FROM sys.master_files

DynamicConnection 8

Step9 : Select Columns from left pane of OLE DB Source Editor and confirm Columns.

DynamicConnection 9

Step10 : Right Click on Data Conversion Task –> Edit and set data types for columns.

DynamicConnection 10

Step11 : Create one database table to store all servers’s database info in central database.


Create Table DatabaseInfo
(
    ServerName Varchar(20),
    DBName varchar(20),
    LogicalName Varchar(200),
    PhysicalName Varchar(200),
    DBSize int
)

DynamicConnection 11

Step12 : Right click on mapping on left pane of OLE DB Destination Editor and map the varialbes to table columns.

DynamicConnection 12

Step13 : Now right click on “DynamicConnection” that you have created before to make it dynamic.

Rikght click on it and click on Property.

Select Expression field button that will open one popup screen.

DynamicConnection 13

Map DynamicConnection’s parameter with varialbes to make it dynamic and run the project.

Regards,

Nirav gajjar

SSIS Package Foreach Loop with ADO Connection

SSIS Package Foreach Loop with ADO Connection.

In my last blog i had written SSIS Package Foreach Loop with File Connection(Link: http://nirav.extreme-advice.com/2013/02/25/ssis-package-foreeach-loop-with-file-connection/).

This package gives you same thing, variable mapping between Foreach Loop Container and any inner task contains in loop (here inner task is Script Task) but here i have used ADO Connection as enumerator in behalf of File Connection.

Step1 : Drag

–> Execute SQL Task to get list of values from database table and import to object variable

–> Foreach Loop Container.

–> Script Task.

ForEachLoopWithADO1

Step2 : Create Variables.

–> Obj_Result – Data Type Object.

–>ServerName – Data Type String.

–> DataSource – Data Type String.

ForEachLoopWithADO2

Step3 : Right Click on Execute SQL Task –> Edit.

–> Result Set – Full Result Text.

–> Connection – Your local database connection.

–> SQL Statement – Select Name,Data_Source From Sys.Servers.

ForEachLoopWithADO3

Step4 : Select Result Set from your left pane of Execute SQL Task Editor.

–> Result Name : 0 (Zero).

–> Variable Name : User::Obj_Result.

ForEachLoopWithADO4

Step5 : Right click on Foreach Loop Container –> Edit.

–> Select Foreach ADO Enumerator.

–> User::Obj_Result.

ForEachLoopWithADO5

Step6 : Select Variable mapping from your left pane of Foreach Loop Editor.

–> Select User::ServerName with Index 0.

–> Select User::DataSource with Index 1.

ForEachLoopWithADO6

Step7 : Rigth click on Script Tak –> Edit.

–> ReadOnlyvariables – Select Both Variable shown in below figure.

ForEachLoopWithADO7

Step8 : Click on Edit Script and paste below code.


#region Namespaces
using System;
using System.Data;
using Microsoft.SqlServer.Dts.Runtime;
using System.Windows.Forms;
#endregion

namespace ST_f67917be72d748f4a19778c4b647dbf0
{
 [Microsoft.SqlServer.Dts.Tasks.ScriptTask.SSISScriptTaskEntryPointAttribute]
 public partial class ScriptMain : Microsoft.SqlServer.Dts.Tasks.ScriptTask.VSTARTScriptObjectModelBase
 {
  public void Main()
  {
            string ServerName, DataSource;
            ServerName = Dts.Variables["User::ServerName"].Value.ToString();
            DataSource = Dts.Variables["User::DataSource"].Value.ToString();
            MessageBox.Show("Server Name : " + ServerName + " Data Source : " + DataSource);
   Dts.TaskResult = (int)ScriptResults.Success;
  }

        #region ScriptResults declaration
        enum ScriptResults
        {
            Success = Microsoft.SqlServer.Dts.Runtime.DTSExecResult.Success,
            Failure = Microsoft.SqlServer.Dts.Runtime.DTSExecResult.Failure
        };
        #endregion

 }
}

This script will show you how to use variables and how to map variables between Execute SQL Task to Foreach Loop Container and Foreach Loop Container to Inner Script Task.

Regards,

Nirav Gajjar

SSIS Package Foreach Loop with File Connection

SSIS Package Foreach Loop with File Connection.

This package gives you an idea to get file details and variable mapping between Foreach Loop Container and any inner task contains in loop (here inner task is Script Task).

Step1 : Drag Foreach Loop and script task from SSIS Toolbox.

ForEachLoopWithFile1

Step2 : Create one variable with name “FileName” and datatype “string”.
this varialbe maps filename from Foreach Loop Container value to Script task.

Step3 : Right click on Foreach Loop Container –> Edit
On Collection pane –> select Foreach File Enumerator in Enumerator field.
choose your folder by clicking on browse button.
–> Select file extention if you want to filter file extention otherwise put *.* to get all files.
–> Select Name and Extention in Retrieve file name.
ForEachLoopWithFile2

Step4 : Select Variable mapping pane on Foreach Loop Editor.
–> Select User::FileName variable.
–> and Index to 0.
ForEachLoopWithFile3

Step5 : Right Click on Script Task –> Edit.
–> Select Script Pane –> Select User::FileName as ReadOnlyVarialbe.

Step6 : Click on Edit Scrit to write script code.

#region Namespaces
using System;
using System.Data;
using Microsoft.SqlServer.Dts.Runtime;
using System.Windows.Forms;
#endregion

namespace ST_f1f3822404bc4e8d9a22fbe2749ff1d9
{
	[Microsoft.SqlServer.Dts.Tasks.ScriptTask.SSISScriptTaskEntryPointAttribute]
	public partial class ScriptMain : Microsoft.SqlServer.Dts.Tasks.ScriptTask.VSTARTScriptObjectModelBase
	{
		public void Main()
		{
            string FileName;
            FileName = Dts.Variables["User::FileName"].Value.ToString();
            MessageBox.Show(FileName);
			Dts.TaskResult = (int)ScriptResults.Success;
		}

        enum ScriptResults
        {
            Success = Microsoft.SqlServer.Dts.Runtime.DTSExecResult.Success,
            Failure = Microsoft.SqlServer.Dts.Runtime.DTSExecResult.Failure
        };
        #endregion

	}
}

Regards,
Nirav gajjar

SSIS Package to get Disk Space Info with WMI and Script Task

SSIS Package to get Disk Space Info with WMI and Script Task.

In Previous blog i have created SSIS Package to get Disk Space Info with WMI.(Link: http://nirav.extreme-advice.com/2013/02/20/ssis-package-to-get-disk-space-info-with-wmi/)
That was a simple way to gather disk space information with importing data in CSV file and exporting data in database.
I have used same table in this task and i have used Script Task in behalf of CSV file.

Table Structure.

Use AdventureWorks2012
Go
Create Table ServerDiskInfo
(
Drive varchar(5),
DriveSize numeric(18,2),
FreeSpace numeric(18,2),
CollectionDate Datetime Not NUll Default GetDate()
)

Lets start to create SSIS Package to get Disk Space Info with WMI and Script Task.

Step1 : Open your SSIS package and drag WMI Data Reader Task from SSIS Toolbox.

DiskSpaceInfoWithWMIScript

Step2 : Create new WMI Connection shown as below.

DiskSpaceInfoWithWMIScript2

Step3 : Create variable Obj_Result with data type object to recieve result data.

DiskSpaceInfoWithWMIScript3

Step4 : Select User::Obj_Result in Destination.

DiskSpaceInfoWithWMIScript4 

Step5 : Right click on Script Task –> Edit.

Select User::Obj_Result as ReadOnlyVariables.

DiskSpaceInfoWithWMIScript5

Step5 : Click on Edit on Script Task Editor and paste below code.


public void Main()
 {
            try
            {
                DataTable DT;
                DataRow DR;
                SqlConnection SQLCon = new SqlConnection("Server=Nirav'sDiary;Database=AdventureWorks2012;User Id=sa;Password=pwd;");

                DT = (DataTable)Dts.Variables["User::Obj_Result"].Value;
                foreach (DataRow row in DT.Rows)
                {
                    DR = row;
                    string Name;
                    int Size, FreeSpace;
                    Name = DR["Name"].ToString();
                    Size = Convert.ToInt32(Convert.ToDouble(DR["Size"].ToString()) / 1024 / 1024 / 1024);
                    FreeSpace = Convert.ToInt32(Convert.ToDouble(DR["FreeSpace"].ToString()) /1024/ 1024 / 1024);
                    SqlCommand cmd = new SqlCommand("Insert Into ServerDiskInfo Values('"+ Name +"',"+ Size +","+ FreeSpace +",GetDate())", SQLCon);
                    SQLCon.Open();
                    cmd.ExecuteNonQuery();
                    SQLCon.Close();
                }

                Dts.TaskResult = (int)ScriptResults.Success;
            }
            catch (Exception e)
            {
                MessageBox.Show(e.Message.ToString());
            }
 }

Step6 : Executing whole task it will make entry in ServerDiskinfo Table.

Query to get ServerDiskInfo.

Declare @Today as datetime
Det @Today = DATEADD(DD,0,DATEDIFF(DD,0,GETDATE()))
Delect Drive,Cast(DriveSize/1024/1024/1024 as Numeric(10,2)) as SizeInGB,Cast(FreeSpace/1024/1024/1024 as Numeric(10,2)) as FreeSpaceInGB from ServerDiskInfo
Where DATEADD(DD,0,DATEDIFF(DD,0,CollectionDate)) = @Today</pre>

Regards,
Nirav gajjar

SSIS Package to get Disk Space Info with WMI

SSIS Package to get Disk Space Info with WMI.

I have created package that gives Disk Space info using WMI Data Reader Task.
You need minor knowledge on WMI Query and WMI Class, here i have used “Win32_LogicalDisk” WMI class.
This package checks disk info for logical disk of your machine and stores value to the database.
To store information create one table on your database.


Use AdventureWorks2012
Go
Create Table ServerDiskInfo
(
Drive varchar(5),
DriveSize numeric(18,2),
FreeSpace numeric(18,2),
CollectionDate Datetime Not NUll Default GetDate()
)

Lets start to create SSIS package.
Step1 : Open your SSIS package and drag WMI Data Reader Task from SSIS Toolbox.
SSIS Package Disk Space With WMI 1

Step2 : Rigth click on Connection Manager pane –> Select New Connection –> Select WMI and click on ADD button.

SSIS Package Disk Space With WMI 2

SSIS Package Disk Space With WMI 3

Step3 : It will opens WMI Connection Manger Editor. you can give any name to connection name and discription.

If you are creating package for your local machine then

\\LocalHost to connect your local machine and check on Use Windows Authentication.

If you are creating package for remote server then

\\RemoteServerName to connect remote server and provide “UserName” and “Password” for User Credentials.

SSIS Package Disk Space With WMI 4

Click on Test button to confirm connetion established successfully.

Step4 : Right Click on WMI Data Reader Task –> Click on Edit –> WMI Options Pane.

Give your WMI Connection –> WMI_localmachine(WMI Connection Manager).

WQL Query Source Type –> Direct Input.

WQL Query Source –>Select Name, Size, FreeSpace From Win32_LogicalDisk Where DriveType = 3.

OUTPUT Type –>DataTable.

OverWrite Destination –> Overwrite Destination.

Destination Type –> File Connecion.

Destination –> Create New Connetion.

SSIS Package Disk Space With WMI 6

Click OK.

SSIS Package Disk Space With WMI 7 New

Click Ok.

Step5 : Execute task to generate C:\ServerDiskInfo.csv file.

Step6: Drag Data Flow task –> Right Click –> Edit.

Step7: Drag Flat File Source from Other Source Tool, Drag Data Conversion from Other Transforms, Drag OLEDB Destination from Other Destinations.

SSIS Package Disk Space With WMI 8 New

Step8 : Right Click on Flat File Source –> Edit

SSIS Package Disk Space With WMI 9

Click on Columns Tab to check the columns then click on OK.

Step9 : Right Click on Data Conversion –> Select FreeSpace and Size.

Give Numeric[DT_numeric] in Data Type to both columns.

SSIS Package Disk Space With WMI 10

Step10 : Right Click on OLE DB Connection –> Edit.

Select OLEDB connection manager. if there is no OLEDB Connection manager then create new connection manager for AdventureWorks2012 and Select ServerDiskInfo Table.

SSIS Package Disk Space With WMI 11

Step11: Select mappings from left pane.

SSIS Package Disk Space With WMI 12 new

map Input Columns with Destination Columns given in image.

Click OK.

Executing whole task it will make entry in ServerDiskinfo Table.

Query to get ServerDiskInfo.

declare @Today as datetime
set @Today = DATEADD(DD,0,DATEDIFF(DD,0,GETDATE()))
select Drive,Cast(DriveSize/1024/1024/1024 as Numeric(10,2)) as SizeInGB,Cast(FreeSpace/1024/1024/1024 as Numeric(10,2)) as FreeSpaceInGB from ServerDiskInfo
where DATEADD(DD,0,DATEDIFF(DD,0,CollectionDate)) = @Today

Regards,
Nirav Gajjar

Script to Remove Jobs History

Script to Remove Jobs History.

You can remove jobs history for particular job for a fixed time.

Below script will remove history data older then 30 days and where job name is “jobname”.

I have used procedure “SP_PURGE_JOBHISTORY” provided by SQL Server, you just need to pass @JobName and @Date as a parameter.


DECLARE @Date DATETIME
DECLARE @Jobname VARCHAR(256)

-- Keep Last 30 days
SET @Date = GETDATE() - 30
SET @Jobname = 'JobName'

EXEC MSDB.DBO.SP_PURGE_JOBHISTORY
@Job_name=@Jobname,
@Oldest_date=@Date

You can also pass JobID instead of JobName.


DECLARE @Date DATETIME
DECLARE @JobID VARCHAR(256)

-- Keep Last 30 days
SET @Date = GETDATE() - 30
SET @Job_Id = 'JobID'

EXEC MSDB.DBO.SP_PURGE_JOBHISTORY
@Job_ID=@Job_ID,
@Oldest_date=@Date

If you want to remove all history for particular job then remove @date from parameter.


DECLARE @JobID VARCHAR(256)
SET @Job_Id = 'JobID'

EXEC MSDB.DBO.SP_PURGE_JOBHISTORY
@Job_name=@Jobname

If you want to clear all history for all jobs just run the procedure “EXEC MSDB.DBO.SP_PURGE_JOBHISTORY”.
it will remove all history for you instance.

SQL Server has two ways to remove data older then 30 days for all jobs.
1) By Script.


DECLARE @Date DATETIME

-- Keep Last 30 days
SET @Date = GETDATE() - 30
SET @Jobname = 'JobName'

EXEC MSDB.DBO.SP_PURGE_JOBHISTORY
@Oldest_date=@Date

2) By SQL Server GUI.
Step1 : Right click on SQL Server Agent and click on property.
Script to remove job history 1

Step2: On the left pane select History and click check box remove agent history.
you have three options day(s), week(s) or month(s) select one of appropriate to remove agent history.

Script to remove job history 2

Regards,

Nirav gajjar

Script to get Jobs Execution Details

Script to get Jobs Execution Details.

This script will give you whole execution details with job started time, job execution time, job finish time, and job execution status.

Note: For multiple scheduled jobs it will fetche only last executed job details.

You need to create one function for that with name “CONV_DATETIME”.

USE [MSDB]
GO
CREATE FUNCTION [DBO].[Conv_datetime](@Id INT,
                                      @It INT)
RETURNS DATETIME
AS
  BEGIN
      DECLARE @Vt CHAR(6),
              @D  CHAR(8),
              @Dt DATETIME

      SELECT
        @D = CONVERT(VARCHAR, @Id)

      IF Len(@It) = 5
        SET @Vt = '0' + CONVERT(CHAR, @It)
      ELSE IF Len(@It) = 4
        SET @Vt = '00' + CONVERT(CHAR, @It)
      ELSE IF Len(@It) = 3
        SET @Vt = '000' + CONVERT(CHAR, @It)
      ELSE IF Len(@It) = 2
        SET @Vt = '0000' + CONVERT(CHAR, @It)
      ELSE IF Len(@It) = 1
        SET @Vt = '00000' + CONVERT(CHAR, @It)

      SELECT
        @Dt = LEFT (@D, 4) + '-' + Substring(@D, 5, 2) + '-'
              + RIGHT(@D, 2) + ' ' + LEFT(@Vt, 2) + ':'
              + Substring(@Vt, 3, 2) + ':' + RIGHT(@Vt, 2)

      RETURN ( @Dt )
  END 

Execute below script after creating the function.

USE MSDB
GO
;WITH JOBSDETAIL
     AS (SELECT
           Row_number()
             OVER(
               PARTITION BY Sj.Name
               ORDER BY Run_time DESC) AS Rnk,
           Sj.Name,
           Run_date,
           Run_time,
           Run_duration                AS Runduration,
           CASE
             WHEN Run_status = 0 THEN 'Failed'
             WHEN Run_status = 1 THEN 'Successes'
             WHEN Run_status = 3 THEN 'Stopped'
           END                         AS Status
         FROM
           DBO.Sysjobhistory Sjh
         JOIN   DBO.Sysjobs Sj
           ON Sjh.Job_id = Sj.Job_id
         WHERE
          Sjh.Step_id = 0
          AND Run_date       = CONVERT(VARCHAR(8), Getdate(), 112))
SELECT
  Name                                                                                                             AS Jobname,
  CONVERT(VARCHAR(5), MSDB.DBO.Conv_datetime(Run_date, Run_time), 108)                                             AS Startdate,
  CONVERT(VARCHAR(5), Stuff(Stuff(RIGHT('000000' + CONVERT(VARCHAR, Runduration), 6), 5, 0, ':'), 3, 0, ':'), 108) AS Runduration,
  CONVERT(VARCHAR(5), Dateadd(SS, ( CASE Len(Runduration)
                                      WHEN 1 THEN Runduration
                                      WHEN 2 THEN Runduration
                                      WHEN 3 THEN ( Cast(LEFT(RIGHT(Runduration, 3), 1) AS INT) * 60 ) + ( RIGHT(Runduration, 2) )
                                      -- min,sec
                                      WHEN 4 THEN ( Cast(LEFT(RIGHT(Runduration, 4), 2) AS INT) * 60 ) + ( RIGHT(Runduration, 2) )
                                      -- min,sec
                                      WHEN 5 THEN ( Cast(LEFT(RIGHT(Runduration, 5), 1) AS INT) * 3600 ) + ( Cast(LEFT(RIGHT(Runduration, 4), 2) AS INT) * 60 ) + RIGHT(Runduration, 2)
                                      WHEN 6 THEN ( Cast(LEFT(RIGHT(Runduration, 6), 2) AS INT) * 3600 ) + ( Cast(LEFT(RIGHT(Runduration, 4), 2) AS INT) * 60 ) + RIGHT(Runduration, 2)
                                    END ), MSDB.DBO.Conv_datetime(Run_date, Run_time)), 108)                       AS Completedat,
  Status,
  Run_date
FROM
  JOBSDETAIL
WHERE
  Rnk = 1 

Regards,
Nirav Gajjar

Script to get SSIS Jobs

Script to get SSIS Jobs.

This script will help you out to get Job Details by category like SSIS,TSQL,etc..


USE MSDB

GO

SELECT Sj.Job_id   AS Jobid,
       Sj.Name     AS Jobname,
       Sjs.Command AS Command
FROM   Sysjobs Sj
       INNER JOIN Sysjobsteps Sjs
               ON( Sj.Job_id = Sjs.Job_id )
WHERE  Sjs.Subsystem = 'SSIS'-- other types (LogReader,PowerShell,Snapshot,TSQL)
       AND Sj.Enabled = 1

GO 

Script to get SSIS Jobs

 

Regards,

Nirav Gajjar

Script to get Job Name from Command

Script to get Job Name from Command

This script helps to get a job name from command or job script.

If you remembered only one word that you have used in your job script and you want to get a job name, this script helps you.

Just paste your word instead of “AnyOneWordFromJobScript”.


USE MSDB
GO

SELECT Sj.Name As JobName,
       St.Command As JobScript
FROM   Sysjobs Sj
       JOIN Sysjobsteps St
         ON Sj.Job_id = St.Job_id
WHERE  St.Command LIKE '%AnyOneWordFromJobScript%' 

Script to get job name from command

 

Regards,

Nirav Gajjar

Script to get failed jobs

Script to get failed jobs
Below is a TSQL query to get failed jobs last hour.
if you want failed jobs for whole day then comment "AND Run_time >= @Ctime" line
DECLARE @Cdate INT
DECLARE @Ctime INT

SET @Cdate = CONVERT(CHAR(8), (SELECT
                                 Dateadd (DAY, ( 0 ), Getdate())), 112)
SET @Ctime = Cast(Replace(CONVERT(CHAR(8), (SELECT
                                              Dateadd (HOUR, -1, Getdate())), 114), ':', '') AS INT);

WITH cte
     AS (SELECT DISTINCT
           Row_number()
             OVER(
               partition BY T2.name
               ORDER BY run_time DESC) AS rnk,
           Substring(T2.Name, 1, 40)   AS 'Job Name',
           T1.Step_id                  AS 'Step_id',
           T1.Step_name                AS 'Step Name',
           T1.Message                  AS 'Error Message',
           Run_time
         FROM
           MSDB.DBO.Sysjobhistory T1
         JOIN   MSDB.DBO.Sysjobs T2
           ON T1.Job_id = T2.Job_id
         WHERE
          T1.Run_status NOT IN ( 1, 4 )
          AND T1.Step_id != 0
          AND Run_date     = @Cdate
          AND Run_time >= @Ctime)
SELECT
  *
FROM
  cte
WHERE
  rnk = 1 

Regards, Nirav gajjar