SSIS Package with Dynamic ADO Connection

SSIS Package with Dynamic ADO Connection.

I have created a package that collects database information from different instances all over the network and stored in one central database table.

For this, I have collected Server Name, Database Name, Logical File Name, Physical File Name and File Size in MB.

I have stored all instance info in my central database table.

Table Script.

Create Table SQLServerInstances
(

    ServerName Varchar(20),
    DBName Varchar(20),

    UserName Varchar(20),

    PWD Varchar(20)
)
Insert Into SQLServerInstances Values('yourLocalMachineName','AdventureWorks2012','sa','password')

 I had taken one Execute SQL Task to get list of instances then move it to Foreach loop and store the values to database table using Data Flow Task.

Step1 : Drag Execute SQL Task, Foreach Loop Container and Data Flow task under Foreach Loop Container.

DynamicConnection 1

Step2 : Create Variables.

–> Obj_Result : DataType Object.

–> ServerName : DataType String.

–> DBName : DataType String.

–> UserName1 : DataType String.

–> PWD : DataType String.

DynamicConnection 2

Step3 : Right click on Execute SQL Task –> Edit and fill the details as shown in below figure.

DynamicConnection 3

Step4 : Select Result Set on left pane of Execute SQL Task and select your object variable.

DynamicConnection 4

Step5 : Right Click on Foreach Loop Container –> Edit.

Select Foreach ADO Enumerator in Enumerator and Select Obj_Result in ADO Object source varialbe field.

DynamicConnection 5

Step6 : Select Variable Mapping on left pane of Foreach Loop Container and select varialbes as you see in below figure.

DynamicConnection 6

Step7 : Now Right click on Data Flow Task –> Edit.

DynamicConnection 7

Step8 : Right click on OLE DB source. create one OLEDB data connection with name Dynamic Connection with your local database connection.

SQL Command.


SELECT @@ServerName as ServerName,
   DB_NAME(database_id) AS DatabaseName,
   Name AS LogicalName ,
   Physical_Name as PhysicalName,
   (size*8)/1024 DBSize
FROM sys.master_files

DynamicConnection 8

Step9 : Select Columns from left pane of OLE DB Source Editor and confirm Columns.

DynamicConnection 9

Step10 : Right Click on Data Conversion Task –> Edit and set data types for columns.

DynamicConnection 10

Step11 : Create one database table to store all servers’s database info in central database.


Create Table DatabaseInfo
(
    ServerName Varchar(20),
    DBName varchar(20),
    LogicalName Varchar(200),
    PhysicalName Varchar(200),
    DBSize int
)

DynamicConnection 11

Step12 : Right click on mapping on left pane of OLE DB Destination Editor and map the varialbes to table columns.

DynamicConnection 12

Step13 : Now right click on “DynamicConnection” that you have created before to make it dynamic.

Rikght click on it and click on Property.

Select Expression field button that will open one popup screen.

DynamicConnection 13

Map DynamicConnection’s parameter with varialbes to make it dynamic and run the project.

Regards,

Nirav gajjar

SSIS Package Foreach Loop with ADO Connection

SSIS Package Foreach Loop with ADO Connection.

In my last blog i had written SSIS Package Foreach Loop with File Connection(Link: http://nirav.extreme-advice.com/2013/02/25/ssis-package-foreeach-loop-with-file-connection/).

This package gives you same thing, variable mapping between Foreach Loop Container and any inner task contains in loop (here inner task is Script Task) but here i have used ADO Connection as enumerator in behalf of File Connection.

Step1 : Drag

–> Execute SQL Task to get list of values from database table and import to object variable

–> Foreach Loop Container.

–> Script Task.

ForEachLoopWithADO1

Step2 : Create Variables.

–> Obj_Result – Data Type Object.

–>ServerName – Data Type String.

–> DataSource – Data Type String.

ForEachLoopWithADO2

Step3 : Right Click on Execute SQL Task –> Edit.

–> Result Set – Full Result Text.

–> Connection – Your local database connection.

–> SQL Statement – Select Name,Data_Source From Sys.Servers.

ForEachLoopWithADO3

Step4 : Select Result Set from your left pane of Execute SQL Task Editor.

–> Result Name : 0 (Zero).

–> Variable Name : User::Obj_Result.

ForEachLoopWithADO4

Step5 : Right click on Foreach Loop Container –> Edit.

–> Select Foreach ADO Enumerator.

–> User::Obj_Result.

ForEachLoopWithADO5

Step6 : Select Variable mapping from your left pane of Foreach Loop Editor.

–> Select User::ServerName with Index 0.

–> Select User::DataSource with Index 1.

ForEachLoopWithADO6

Step7 : Rigth click on Script Tak –> Edit.

–> ReadOnlyvariables – Select Both Variable shown in below figure.

ForEachLoopWithADO7

Step8 : Click on Edit Script and paste below code.


#region Namespaces
using System;
using System.Data;
using Microsoft.SqlServer.Dts.Runtime;
using System.Windows.Forms;
#endregion

namespace ST_f67917be72d748f4a19778c4b647dbf0
{
 [Microsoft.SqlServer.Dts.Tasks.ScriptTask.SSISScriptTaskEntryPointAttribute]
 public partial class ScriptMain : Microsoft.SqlServer.Dts.Tasks.ScriptTask.VSTARTScriptObjectModelBase
 {
  public void Main()
  {
            string ServerName, DataSource;
            ServerName = Dts.Variables["User::ServerName"].Value.ToString();
            DataSource = Dts.Variables["User::DataSource"].Value.ToString();
            MessageBox.Show("Server Name : " + ServerName + " Data Source : " + DataSource);
   Dts.TaskResult = (int)ScriptResults.Success;
  }

        #region ScriptResults declaration
        enum ScriptResults
        {
            Success = Microsoft.SqlServer.Dts.Runtime.DTSExecResult.Success,
            Failure = Microsoft.SqlServer.Dts.Runtime.DTSExecResult.Failure
        };
        #endregion

 }
}

This script will show you how to use variables and how to map variables between Execute SQL Task to Foreach Loop Container and Foreach Loop Container to Inner Script Task.

Regards,

Nirav Gajjar

SSIS Package Foreach Loop with File Connection

SSIS Package Foreach Loop with File Connection.

This package gives you an idea to get file details and variable mapping between Foreach Loop Container and any inner task contains in loop (here inner task is Script Task).

Step1 : Drag Foreach Loop and script task from SSIS Toolbox.

ForEachLoopWithFile1

Step2 : Create one variable with name “FileName” and datatype “string”.
this varialbe maps filename from Foreach Loop Container value to Script task.

Step3 : Right click on Foreach Loop Container –> Edit
On Collection pane –> select Foreach File Enumerator in Enumerator field.
choose your folder by clicking on browse button.
–> Select file extention if you want to filter file extention otherwise put *.* to get all files.
–> Select Name and Extention in Retrieve file name.
ForEachLoopWithFile2

Step4 : Select Variable mapping pane on Foreach Loop Editor.
–> Select User::FileName variable.
–> and Index to 0.
ForEachLoopWithFile3

Step5 : Right Click on Script Task –> Edit.
–> Select Script Pane –> Select User::FileName as ReadOnlyVarialbe.

Step6 : Click on Edit Scrit to write script code.

#region Namespaces
using System;
using System.Data;
using Microsoft.SqlServer.Dts.Runtime;
using System.Windows.Forms;
#endregion

namespace ST_f1f3822404bc4e8d9a22fbe2749ff1d9
{
	[Microsoft.SqlServer.Dts.Tasks.ScriptTask.SSISScriptTaskEntryPointAttribute]
	public partial class ScriptMain : Microsoft.SqlServer.Dts.Tasks.ScriptTask.VSTARTScriptObjectModelBase
	{
		public void Main()
		{
            string FileName;
            FileName = Dts.Variables["User::FileName"].Value.ToString();
            MessageBox.Show(FileName);
			Dts.TaskResult = (int)ScriptResults.Success;
		}

        enum ScriptResults
        {
            Success = Microsoft.SqlServer.Dts.Runtime.DTSExecResult.Success,
            Failure = Microsoft.SqlServer.Dts.Runtime.DTSExecResult.Failure
        };
        #endregion

	}
}

Regards,
Nirav gajjar

SSIS Package to get Disk Space Info with WMI and Script Task

SSIS Package to get Disk Space Info with WMI and Script Task.

In Previous blog i have created SSIS Package to get Disk Space Info with WMI.(Link: http://nirav.extreme-advice.com/2013/02/20/ssis-package-to-get-disk-space-info-with-wmi/)
That was a simple way to gather disk space information with importing data in CSV file and exporting data in database.
I have used same table in this task and i have used Script Task in behalf of CSV file.

Table Structure.

Use AdventureWorks2012
Go
Create Table ServerDiskInfo
(
Drive varchar(5),
DriveSize numeric(18,2),
FreeSpace numeric(18,2),
CollectionDate Datetime Not NUll Default GetDate()
)

Lets start to create SSIS Package to get Disk Space Info with WMI and Script Task.

Step1 : Open your SSIS package and drag WMI Data Reader Task from SSIS Toolbox.

DiskSpaceInfoWithWMIScript

Step2 : Create new WMI Connection shown as below.

DiskSpaceInfoWithWMIScript2

Step3 : Create variable Obj_Result with data type object to recieve result data.

DiskSpaceInfoWithWMIScript3

Step4 : Select User::Obj_Result in Destination.

DiskSpaceInfoWithWMIScript4 

Step5 : Right click on Script Task –> Edit.

Select User::Obj_Result as ReadOnlyVariables.

DiskSpaceInfoWithWMIScript5

Step5 : Click on Edit on Script Task Editor and paste below code.


public void Main()
 {
            try
            {
                DataTable DT;
                DataRow DR;
                SqlConnection SQLCon = new SqlConnection("Server=Nirav'sDiary;Database=AdventureWorks2012;User Id=sa;Password=pwd;");

                DT = (DataTable)Dts.Variables["User::Obj_Result"].Value;
                foreach (DataRow row in DT.Rows)
                {
                    DR = row;
                    string Name;
                    int Size, FreeSpace;
                    Name = DR["Name"].ToString();
                    Size = Convert.ToInt32(Convert.ToDouble(DR["Size"].ToString()) / 1024 / 1024 / 1024);
                    FreeSpace = Convert.ToInt32(Convert.ToDouble(DR["FreeSpace"].ToString()) /1024/ 1024 / 1024);
                    SqlCommand cmd = new SqlCommand("Insert Into ServerDiskInfo Values('"+ Name +"',"+ Size +","+ FreeSpace +",GetDate())", SQLCon);
                    SQLCon.Open();
                    cmd.ExecuteNonQuery();
                    SQLCon.Close();
                }

                Dts.TaskResult = (int)ScriptResults.Success;
            }
            catch (Exception e)
            {
                MessageBox.Show(e.Message.ToString());
            }
 }

Step6 : Executing whole task it will make entry in ServerDiskinfo Table.

Query to get ServerDiskInfo.

Declare @Today as datetime
Det @Today = DATEADD(DD,0,DATEDIFF(DD,0,GETDATE()))
Delect Drive,Cast(DriveSize/1024/1024/1024 as Numeric(10,2)) as SizeInGB,Cast(FreeSpace/1024/1024/1024 as Numeric(10,2)) as FreeSpaceInGB from ServerDiskInfo
Where DATEADD(DD,0,DATEDIFF(DD,0,CollectionDate)) = @Today</pre>

Regards,
Nirav gajjar

SSIS Package to get Disk Space Info with WMI

SSIS Package to get Disk Space Info with WMI.

I have created package that gives Disk Space info using WMI Data Reader Task.
You need minor knowledge on WMI Query and WMI Class, here i have used “Win32_LogicalDisk” WMI class.
This package checks disk info for logical disk of your machine and stores value to the database.
To store information create one table on your database.


Use AdventureWorks2012
Go
Create Table ServerDiskInfo
(
Drive varchar(5),
DriveSize numeric(18,2),
FreeSpace numeric(18,2),
CollectionDate Datetime Not NUll Default GetDate()
)

Lets start to create SSIS package.
Step1 : Open your SSIS package and drag WMI Data Reader Task from SSIS Toolbox.
SSIS Package Disk Space With WMI 1

Step2 : Rigth click on Connection Manager pane –> Select New Connection –> Select WMI and click on ADD button.

SSIS Package Disk Space With WMI 2

SSIS Package Disk Space With WMI 3

Step3 : It will opens WMI Connection Manger Editor. you can give any name to connection name and discription.

If you are creating package for your local machine then

\\LocalHost to connect your local machine and check on Use Windows Authentication.

If you are creating package for remote server then

\\RemoteServerName to connect remote server and provide “UserName” and “Password” for User Credentials.

SSIS Package Disk Space With WMI 4

Click on Test button to confirm connetion established successfully.

Step4 : Right Click on WMI Data Reader Task –> Click on Edit –> WMI Options Pane.

Give your WMI Connection –> WMI_localmachine(WMI Connection Manager).

WQL Query Source Type –> Direct Input.

WQL Query Source –>Select Name, Size, FreeSpace From Win32_LogicalDisk Where DriveType = 3.

OUTPUT Type –>DataTable.

OverWrite Destination –> Overwrite Destination.

Destination Type –> File Connecion.

Destination –> Create New Connetion.

SSIS Package Disk Space With WMI 6

Click OK.

SSIS Package Disk Space With WMI 7 New

Click Ok.

Step5 : Execute task to generate C:\ServerDiskInfo.csv file.

Step6: Drag Data Flow task –> Right Click –> Edit.

Step7: Drag Flat File Source from Other Source Tool, Drag Data Conversion from Other Transforms, Drag OLEDB Destination from Other Destinations.

SSIS Package Disk Space With WMI 8 New

Step8 : Right Click on Flat File Source –> Edit

SSIS Package Disk Space With WMI 9

Click on Columns Tab to check the columns then click on OK.

Step9 : Right Click on Data Conversion –> Select FreeSpace and Size.

Give Numeric[DT_numeric] in Data Type to both columns.

SSIS Package Disk Space With WMI 10

Step10 : Right Click on OLE DB Connection –> Edit.

Select OLEDB connection manager. if there is no OLEDB Connection manager then create new connection manager for AdventureWorks2012 and Select ServerDiskInfo Table.

SSIS Package Disk Space With WMI 11

Step11: Select mappings from left pane.

SSIS Package Disk Space With WMI 12 new

map Input Columns with Destination Columns given in image.

Click OK.

Executing whole task it will make entry in ServerDiskinfo Table.

Query to get ServerDiskInfo.

declare @Today as datetime
set @Today = DATEADD(DD,0,DATEDIFF(DD,0,GETDATE()))
select Drive,Cast(DriveSize/1024/1024/1024 as Numeric(10,2)) as SizeInGB,Cast(FreeSpace/1024/1024/1024 as Numeric(10,2)) as FreeSpaceInGB from ServerDiskInfo
where DATEADD(DD,0,DATEDIFF(DD,0,CollectionDate)) = @Today

Regards,
Nirav Gajjar

SSIS Package to Resize image

SSIS Package to Resize image.

This script helps you to resize images as you needed in percentage(Dynamically).

Step1 : drag n drop foreeach loop container.

Step2 : drag n drop script task in to foreach loop container.

Step3 : Right click on any part of your control flow and select variables to see list of variables.

Create two new string variable with name “ImageFileName” and “ImagePath” and assign your local image folder path to ImagePath Variable value.

Note: Here local path is “D:\Work\Images”.

[image5.png]

Step4: Right click on foreeachloop container and click on edit.Select Collection from left pane.

Select Foreeach File Enumerator in Enumerator and select your variable “@[User:ImagePath]” in directory and click ok.

SSIS Resize Image 2

Step5 : Go to variable mappings and select imagefilename variable with index 0 and click ok.

SSIS Resize Image 3

Step6: Right click on script task and click on edit.

Select Script from left pane.

Select variable imagefilename for readonlyvariables.

SSIS Resize Image 4

Step7: Click on Edit Script and paste below code.

#region Help:  Introduction to the script task

/* The Script Task allows you to perform virtually any operation that can be accomplished in

  * a .Net application within the context of an Integration Services control flow.

  *

  * Expand the other regions which have "Help" prefixes for examples of specific ways to use

  * Integration Services features within this script task. */

 #endregion

 #region Namespaces

 using System;

 using System.Data;

 using System.Drawing;

 using System.Drawing.Drawing2D;

using System.Drawing.Imaging;

using Microsoft.SqlServer.Dts.Runtime;

using System.Windows.Forms;

#endregion

namespace ST_f8e4dbfd074e4f9296466d7c3bc1f83d

{

    ///

    /// ScriptMain is the entry point class of the script.  Do not change the name, attributes,

    /// or parent of this class.

    ///
   [Microsoft.SqlServer.Dts.Tasks.ScriptTask.SSISScriptTaskEntryPointAttribute]

    public partial class ScriptMain : Microsoft.SqlServer.Dts.Tasks.ScriptTask.VSTARTScriptObjectModelBase

  {

       public void Main()

       {

             try

             {// TODO: Add your code here

                 string ImageFileName;

                 ImageFileName = Dts.Variables["ImageFileName"].Value.ToString();

                 // Get the scale factor.

                 using (Image img = Image.FromFile(ImageFileName))

                 {

                     int percent = 25;

                     //int i;

                     //for (i= img.Height; i> 100; i= (img.Height * 90) / 100)

                     //{

                     //    percent = percent - 10;

                     //}

                     int width = Convert.ToInt32(img.Width * (percent * 0.01));

                     int height = Convert.ToInt32(img.Height * (percent * 0.01));

                     ImageFormat format = img.RawFormat;

                     Image thumbNail = new Bitmap(width, height, img.PixelFormat);

                     Graphics g = Graphics.FromImage(thumbNail);

                     g.CompositingQuality = CompositingQuality.HighQuality;

                     g.SmoothingMode = SmoothingMode.HighQuality;

                     g.InterpolationMode = InterpolationMode.HighQualityBicubic;

                     Rectangle rect = new Rectangle(0, 0, width, height);

                     g.DrawImage(img, rect);                   

                     img.Dispose();

                     thumbNail.Save(ImageFileName, format);

                 }

                 Dts.TaskResult = (int)ScriptResults.Success;

             }

             catch (Exception ex)

             {

                 MessageBox.Show(ex.Message.ToString());

             }

      }

         #region ScriptResults declaration

         ///

        /// This enum provides a convenient shorthand within the scope of this class for setting the

        /// result of the script.

        ///

        /// This code was generated automatically.

        ///
        enum ScriptResults

        {

            Success = Microsoft.SqlServer.Dts.Runtime.DTSExecResult.Success,

            Failure = Microsoft.SqlServer.Dts.Runtime.DTSExecResult.Failure

        };

        #endregion

  }

}

Regards,
Nirav Gajjar

SSIS Package to Read IISLog

SSIS Package to Read IISLog

You need LogParser 2.2 installed on your machine.

I need iislog every 1 hour so i have set this package to execute every hour and imports second last files to database.
Before you start creating package you have to create one batch file and one table.

Step1 : First of all you have to create batch file with below script and save it to E:\BatchFiles\Process_IISLog.bat

"c:\program files (x86)\log parser 2.2\LogParser.exe" "SELECT s-computername,LogFileName ,rOWnUMBER,S-SITENAME,S-IP,CS-METHOD,CS-URI-STEM,CS-URI-QUERY,S-PORT,CS-USERNAME,C-IP, SC-STATUS,SC-SUBSTATUS, SC-WIN32-STATUS,SC-BYTES,CS-BYTES,TIME-TAKEN INTO TestDB1.dbo.IISLogDetails FROM %1" -i:W3C -o:SQL -server:ExpertsClick -database:testdb1 -username:sa -password:pwd

Step2 : Create table to store IIS Log.

USE [TestDB1]
GO
/****** Object: Table [dbo].[IISLogDetails] Script Date: 07-02-2013 21:20:19 ******/
SET ANSI_NULLS ON
GO

SET QUOTED_IDENTIFIER ON
GO

SET ANSI_PADDING ON
GO

CREATE TABLE [dbo].[IISLogDetails](
[HostName] [varchar](25) NULL,
[LogFileName] [varchar](150) NULL,
[RowNumber] [int] NULL,
[LogFolder] [varchar](25) NULL,
[IP] [varchar](25) NULL,
[HTTP_Method] [varchar](5) NULL,
[WebURL] [varchar](100) NULL,
[QueryString] [varchar](250) NULL,
[Port] [int] NULL,
[UserName] [varchar](25) NULL,
[ClientIP] [varchar](25) NULL,
[StatusNumber] [int] NULL,
[SubStatus] [int] NULL,
[Win32Status] [int] NULL,
[SCBytes] [int] NULL,
[CSBytes] [int] NULL,
[TimeTaken] [int] NULL
) ON [PRIMARY]
GO

SET ANSI_PADDING OFF
GO

Step3 : Create new integration service project with name “ReadIISLog”.

Step4 : Drag Foreeach Loop Container from toolbox.

Step5 : Drag script task under Foreeach Loop Container.

Step6 : Create two variables named “LogFileName” and “MainLogPath” with data type string.

5

Step7 : assign path to MainLogPath variable for iislog.
default path is “C:\WINDOWS\system32\LogFiles\W3SVC456758617”

Step8 : Right click on Foreach Loop Container and click on Edit.

1

Step9 : Click on Expressions button and select Directory in Property column and drag MainLogPath variable in Expression.

2

Step10 : Click OK on Property Expressions Editor.

Step11 : Select Variable Mapping in left penal of Foreeach Loop Editor and choose User::LogFileName in Variable and 0 as Index.

3

Step12: Select Script task editor, right click on it and click on Edit. Select User::LogFileName as ReadOnlyVariables.

4

Step13: Click on Edit Script and Paste Below Code.

.
/*
Microsoft SQL Server Integration Services Script Task
Write scripts using Microsoft Visual C# 2008.
The ScriptMain is the entry point class of the script.
*/
using System;
using System.Data;
using System.Diagnostics;
using System.IO;
using System.Text;
using System.Data.SqlClient;
using Microsoft.SqlServer.Dts.Runtime;
using System.Windows.Forms;

namespace ST_42efca1c007548f2bf3c1dfc977efd42.csproj
{
[Microsoft.SqlServer.Dts.Tasks.ScriptTask.SSISScriptTaskEntryPointAttribute]
public partial class ScriptMain : Microsoft.SqlServer.Dts.Tasks.ScriptTask.VSTARTScriptObjectModelBase
{
#region VSTA generated code
enum ScriptResults
{ <code>
 Success = Microsoft.SqlServer.Dts.Runtime.DTSExecResult.Success,</code>
Failure = Microsoft.SqlServer.Dts.Runtime.DTSExecResult.Failure
};
#endregion
public void Main()
{
try
{
string LogFileName = Dts.Variables["User::LogFileName"].Value.ToString();
FileInfo LogFile = new FileInfo(LogFileName);
DateTime LogFileDate = LogFile.LastWriteTime;
TimeSpan timespan = (DateTime.Now - LogFileDate);
if (timespan.Hours >= 1)
{
//From Command Line.
System.Diagnostics.Process proc = new System.Diagnostics.Process();
proc.StartInfo.FileName = "E:\\BatchFiles\\Process_IISLog.bat";
proc.StartInfo.Arguments = LogFile.FullName.ToString();
proc.StartInfo.RedirectStandardError = false;
proc.StartInfo.RedirectStandardOutput = false;
proc.StartInfo.UseShellExecute = false;
proc.Start();
proc.WaitForExit();
//Move Imported file to oldlog folder.
File.Move(LogFileName, LogFile.DirectoryName.ToString() + "\\oldLog\\" + LogFile.Name.ToString());
}
Dts.TaskResult = (int)ScriptResults.Success;
}
catch (Exception ex)
{
MessageBox.Show("Error : " + ex.Message.ToString());
}
}
}
}

Regards,
Nirav Gajjar